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1  Oth  Annual  Systems  Engineering  Conference 
“Focusing  on  Improving  Performance  of  Defense  Systems  Programs ” 

San  Diego,  CA 
22-25  October  2007 
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Agenda 


Tuesday.  23  October  2007 

Keynote  Addresses: 

•  Hon  James  Finley.  Deputy  Under  Secretary  of  Defense,  Acquisition  and  Technology 

•  Hon  Charles  McOueary.  Director,  Operational  Test  and  Evaluation 

Plenary  Session:  Executive  Panel: 

•  Mr.  Mark  Schaeffer.  Director,  Office  of  Under  Secretary  of  Defense  of  Acquisition,  Technology  and  Logistics;  Director,  Systems  and  Software  Engineering 

Panelists: 

•  Mr.  Terry  Jaggers .  SAF/AQR  -  Science,  Technology,  and  Engineering 

•  Mr.  Carl  Siel.  Chief  Engineer,  Office  of  the  Assistant  Secretary  of  the  Navy  Research,  Development  and  Acquisition 

•  Mr.  Doug  Wiltsie.  HODA.  OASA  (ALT) 


Track  1 

•  “Update:  OSD  Systems  Engineering  Revitalization  Efforts,”  Col  Richard  Hoeferkamp,  USAF 

•  “The  Effectiveness  of  Systems  Engineering:  On  Federal  (DoD)  System  Development  Programs”,  Mr.  A1  Mink,  SRA  International 

•  “Tools  and  Resources  to  Enable  Systems  Engineering  Improvement,”  Mr.  Michael  Kutch,  SPAWAR  Systems  Center  Charleston 

•  “Sound  Systems  Engineering  Assures  Proper/Early  Producibility”,  Dr.  Thomas  Christian,  Aeronautical  Systems  Center 

•  “Realization  of  Systems  Engineering  For  the  Future”,  Ms.  Karen  Bausman,  AF  Center  for  Systems  Engineering 

Track  2 

•  “Developmental  Test  &  Evaluation  Policy  Vectors”,  Ms.  Darlene  Mosser-Kemer  OUSD  (AT&L) 

•  “Test  Strategy  Done  Early  Drives  Test  Planning  and  Successful  Testing”,  Mr.  William  Lyders,  AS  SETT,  Inc. 

•  “Applying  Design  of  Experiments  Methodology  to  Sortie  Generation  Rate  Evaluation”,  Mr.  Joseph  Tribble, _AVW_“Implementing  a  Systems  Engineering 
Risk  Program  in  a  Sustainment  Environment”,  Mr.  James  Miller,  USAF 

•  “Joint  Safety  Review  Process  Study”,  Ms.  Paige  Ripini,  Booz  Allen  Hamilton 

Track  3 

•  “DoD  Systemic  Root  Cause  Analysis”,  Mr.  Dave  Castellano,  OUSD  (AT&L) 

•  “Applying  Systems  Engineering  During  Pre-Acquisition  Activities”,  Lt  Col  Mark  Wilson,  USAF 

•  “Reforming  the  DoD  Acquisition  Process — A  Systems  Engineering  Approach”,  Mr.  Stephan  Ward,  U.S.  Air  Force 

•  “The  Effectiveness  of  Systems  Engineering:  On  Federal  System  Development  Programs”,  Mr.  Alan  Mink,  SRA  International 

Track  4 

•  “Environment,  Safety,  and  Occupational  Health  (ESOH) — Design  Considerations  to  Support  Sustainability  and  Readiness”,  Ms.  Patricia  Huheey,  ODUSD 
(I&E) 

•  “Real-Time  Diagnostics  for  High  Availability  Systems”,  Mr.  Edward  Beck,  Computer  Sciences  Corporation 

•  “Sparing  Satellites  Comparative  Strategies  of  On-orbit  &  In-factory  Storage”,  Mr.  James  Mazzei,  The  Aerospace  Corporation 

Track  5 

•  “Acquisition  M&S  Master  Plan  Implementation  Status”,  Mr.  Michael  Truelove,  SAIC 

•  “Establish  M&S-Related  Guidelines  for  Solicitations,  Source  Selections,  and  Contracting”,  Mr.  Michael  Truelove,  SAIC 

•  “Modeling  and  Simulation  Resource  Reuse  Business  Model”,  Mr.  Dennis  Shea,  Center  for  Naval  Analyses 
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•  “Modeling  and  Simulation  Support  Plan”,  Mr.  David  Henry,  Lockheed  Martin 

•  “Modeling  and  Simulation  Education  for  the  Acquisition/T&E  Workforce:  Requirements  Analysis”,  Mr.  David  Olwell,  NPS 
Track  6 

•  “The  Use  of  Enterprise  Architecture  to  Support  the  Development  of  the  Next  Generation  Air  Transportation  System  (NextGen)”,  Mr.  David  Synder,  The 
MITRE  Corp. 

•  “Complex  Systems  of  Systems:  The  Dual  Challenge”,  Mr.  Phillip  Boxer,  Software  Engineering  Institute 

•  “Systems  Engineering  in  the  Cognitive  and  Social  Domains  of  Net  Centric  Operations”,  Dr.  Abe  Meilich,  Lockheed  Martin 


Track  7 

•  “Simplifying  &  Scaling  Engineering  Processes:  Unifying  Business  Units  and  Engineering  Disciplines”,  Mr.  Raymond  Jorgensen,  Rockwell  Collins 

•  “NAVAIR  Systems  Engineering  Revitalization”,  Mr.  Michael  Gaydar,  Department  of  Navy,  NAVAIR 

•  “Integrating  Engineering  Project  Management  and  Product  Development  Processes”,  Mr.  Raymond  Jorgensen,  Rockwell  Collins 

•  “Engineering  for  System  Assurance — Legacy,  Life  Cycle,  Leadership”,  Mr.  Paul  Croll,  Computer  Sciences  Corporation 

Track  8 

o  “DoD  Software  Engineering  and  System  Assurance”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  Systems  and  Software  Engineering 
o  “The  Integrated  Software  and  Systems  Engineering  Curriculum  Project:  Creating  a  Reference  Curriculum  for  Graduate  Software  Engineering 
Education”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  Systems  and  Software 
o  “Requirements  for  a  Chief  Software  Engineer  in  a  DoD  Acquisition  Agency”,  Mr.  A1  Florence,  The  MITRE  Corporation 
o  “Developing  an  Integrated  Process  Methodology  for  Interim  Software  Releases”,  Mr.  Tim  Woods,  Southern  Methodist  University 

Wednesday.  24  October  2007 
Track  1 

o  “Change  Management  of  UML-Based  Systems  Engineering  Artefacts”,  Mr.  David  Price,  Eurostep 
o  “A  Day  in  the  Life  of  a  Verification  Requirement”,  Mr.  Stephen  Scukanec,  Northrop  Grumman  Corporation 
o  “How  to  Measurably  Improve  Your  Requirements”,  Mr.  Timothy  Olson,  Lean,  Solutions  Institute,  Inc. 
o  “Case  Studies:  A  Common  Language  Between  Engineers  and  Managers”,  Capt  DeWitt  Latimer,  USAF 
o  “A  Strategy  for  Improved  System  Assurance”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 

°  “Discussion  of  the  U.S.  Army  RDECOM  APS  Objective  Trade  Study”,  Mr.  Frank  Salvatore,  High  Performance  Technologies,  Inc. 
o  “Program  Support  Review  Deep  Dive”,  Mr.  Peter  Nolte,  OUSD/SSE 

Track  2 

o  “An  Update  on  the  DT&E  Committee’s  Recommended  Policy  Changes  to  DoD  5000”,  Col  Richard  Stuckey,USAF,OUSD  (AT&L)/SSE/  DT&E 
o  “System  Test  and  Evaluation  in  the  DARPA  Immune  Building  Demonstration  Program”,  Mr.  Mark  Saxon,Battelle 

o  “  Modeling  and  Simulation  in  the  Navy  Warfare  Systems  Test  &  Evaluation  Enterprise”,  Ms.  Shala  Malone, Navy  Program  Executive  Office  Integrated 
Warfare 

o  “Joint  Mission  Environment  Test  Capability  (JMETC)”,  Mr.  Richard  Lockhart,  Test  Resource  Management  Center 
o  “Testing  Concept  of  Operations  in  DoD’s  Net  Centric  Environment”,  Mr.  Steve  Reeder,  South  Carolina 
o  Research  Authority  (SCRA 

o  “Do  it  right,  do  it  early;  Do  it  early,  do  it  right” — Considerations  for  the  Early  Stages  of  Concept,  System,  and  Systems  of-Systems  Definition”,  Mr.  Jeff 
Loren,  MTC  Technologies,  Inc.  (SAF/AQRE) 

o  “Applications  of  Systems  Engineering  to  Pre-Milestone  A  Projects”,  Ms.  Lori  Zipes, Naval  Surface  Warfare  Center  PC 
o  “Systems  Engineering  in  a  Systems  of  Systems  Environment  -  Defense  Update”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 
o  “ASW  System-of- Systems  Engineering  Analysis  Applied  in  an  LCS  ASW  Integration  Pilot  Project”,  Mr.  G.  Richard  Thompson,  JHU/APL 


Track  3 

o  “Systems  Engineering  Plan  Preparation  Guide  Update”,  Mr.  Chester  Bracuto,  OSD/AT&L/A&T/SSE/ED 
o  “  Toward  a  Unified  Systems  Engineering  Plan”,  Mr.  Robert  Scheurer,  Boeing  Integrated  Defense  Systems 
o  “Integrating  Risk  &  Knowledge  Management”,  Mr.  David  Lengyel,  NASA 

o  “Systems  Engineering  and  Program  Management — How  Different  are  They?”,  Ms.  Lori  Zipes,  Naval  Surface  Warfare  Center  PC 
o  “Systems  Engineering  Analysis  to  Improve  Concept  Development  of  Complex  Defense  Systems”,  Mr.  Michael  Harper,  SPAWAR  Systems  Center 
Charleston 

o  “  The  Joint  Partnership  Between  Program  Management  and  Systems  Engineering”,  Mr.  Samuel  Son,  The  Boeing  Company 
o  “Lockheed  Martin  Aeronautics  Company  Approach  to  Solving  Systemic  Development  Program  Issues”,  Mr.  John  Weaver,  Lockheed  Martin 
Aeronautics  Company 

o  “Airborne  Signals  Intelligence  Payload  (ASIP)  Program  Integrated  Risk  Management”,  Mr.  Danial  Bolek,  USAF 
o  “Improvements  to  the  Risk  Management  Process”,  Mr.  Doug  Atkinson,  USAF 
o  “Integrated  Risk  and  Earned  Value  Management”,  Mr.  Paul  Davis,  Northrop  Grumman 

o  “Application  of  Risk  Management  Practices  to  NNSA  Tritium  Readiness  Subprogram”, Mr.  Sham  Shete,  Washington  Savannah  River  Co. 

Track  4 

o  “Defining  Lean  Service  and  Maintenance  Processes”,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Inc. 

o  “Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the  U.S.  Coast  Guard”,  Mr.  Patrick  Cumby,  VectorCSP,  L 
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“Asset-Based  PBL  for  Navy  Warships  -  A  Case  Study  for  LCS  Class  Ships”,  Mr.  Michael  Mahon,  Lockheed  Martin 
o  “Integrated  Diagnostics  Closed  Loop  Data  System  (At  the  Point  of  Use)  (Support  Systems  Knowledge  Engineering  Enhances  Traditional  Support 
Equipment  Systems  Engineering),”  Mr.  Steven  Head,  Boeing 
o  “Aging  Aircraft  Sustainment  with  Non-Standard  Engineering”,  Ms.  Kendal  Hinton,  Georgia  Tech  Research  Institute 
o  “Maintaining  System  Viability  for  the  Long  Term”,  Mr.  Peter  Henry,  BAE  Systems  Land  and  Armament 
o  “C-17  Program  Applies  Systems  Engineering  to  a  Large  Improvement  Project”,  Mr.  Brent  Theodore, The  Boeing  Company 

Track  5 

o  “ Advancing  the  FEDEP  for  Simulation  Based  Acquisition ”,  Dr.  Katherine  Morse,  SAIC 

o  “Acquisition  M&S  Community  Sponsored  M&S  Project:  Standardized  Documentation  for  Verification,  Validation,  and  Accreditation”,  Mr.  Kevin 
Charlow,  (paper)  (slides)  Space  and  Naval  Warfare  Systems  Center  Charleston 
o  “A  Methodology  for  Assessing  &  Prioritizing  the  Risks  Associated  with  the  Level  of  Verification,  Validation  and  Accreditation  (VV&A)  of  Models 
and  Simulations”,  Dr.  James  Elele,  U.S.  Navy 

o  “Experiences  in  Applying  SysML  to  Develop  Interoperable  Torpedo  Modeling  and  Simulation  Components”,  Mr.  Thomas  Haley,  Naval  Undersea 
Warfare  Center 

o  “Unifying  Systems  Engineering  Simulations”,  Mr.  Ryan  O’ Grady,  Cybernet  Systems  Corporation 
o  “Information  Modeling  for  Systems  Integration”,  Ms.  Claudia  Rose,BBII 

o  “Simulation  Supported  Decision  Making”,  (slidesl)  (slides  2)  Mr.  Gene  Allen,  MSC  Software  Corporation 
Track  6 

o  “Achieving  Agility  in  Cyberspace”,  Mr.  Phillip  Boxer,  Software  Engineering  Institute 
o  “Application  of  Autonomic  Agents  for  Global  Information  Grid”,  Mr.  David  Cox,  University  of  Arizona 
o  “Architecture-Based  Concept  Evaluation  in  Support  of  JCIDS”,  Dr.  David  Jacques,  Air  Force  Institute  of  Technology 
o  “System  of  Systems  Implications  for  Operational  Test”,  Dr.  John  Colombi,  Air  Force  Institute  of  Technology 
°  “Case  Study:  Net  Centric  Mission  Thread  Modeling  and  Analysis”,  Dr.  Prem  Jain,  MITRE 

o  “Quantitative  Comparison  of  Alternative  Designs  for  a  JC3M  System”,  Mr.  Gregory  Miller,  Naval  Postgraduate  School 
o  “Advanced  Net  Centric  Simulation  for  Aerospace  Command  and  Control”,  Ms.  Kimberly  Kendall,  753d  ELSG/NEM,  ESC,  USAF 


Track  7 

o  “CMMI— Next  Steps”,  Ms.  Kristen  Baldwin,  ODUSD  (A&T)  SSE/SSA 

o  “CMMI  Instructional  Challenges  to  Systems  Engineers  in  Small  Settings”,  Dr.  Mary  Anne  Herndon,  Transdyne  Corporation 
“FISMA  Operational  Controls  and  Their  Relationship  to  Process  Maturity”,  Ms.  Rhonda  Henning,  Harris  Corporation 

o  “Executing  a  Successful  CMMI  Maturity  Level  3  SCAMPI  For  SPAWAR  Systems  Center  Charleston”,  Mr.  Michael  Kutch,  SPAWAR  Systems  Center 
Charleston 

o  “CMMI  for  Services:  Re-Introducing  the  CMMI  for  Services  Constellation”,  Mr.  Craig  Hollenbach,  Northrop  Grumman  Corporation 

o  “How  to  Paint  a  Room:  The  Role  of  Specs  &  Standards  in  SE”,  Mr.  Robert  Kuhnen,  USAF 

o  “Continuous  Improvement  at  the  Organization,  Team,  and  Individual  Levels  — Lessons  Learned  Integrating  CMM,TSP,  and  PSP  and  Why  All  Three 
are  Needed”,  Mr.  Girish  Seshagiri, Advanced  Information  Services,  Inc 

o  “Addressing  Environment,  Safety  and  Occupational  Health  Issues  for  the  Mine  Resistant  Ambush  Protected  (MRAP)  Vehicle  Program”,  Ms.  Jennifer 
Malone,  EG&G 

o  “Anatomy  of  an  Award  Winning  Safety  Program:  A  Case  Study  of  the  SSGN  OHIO  Class  Conversion  Safety  Program”,  Mr.  Ricky  Milnarik,  Electric 
Boat  Corporation 

o  “The  Safety  of  Unmanned  Systems:  The  Process  Used  to  Develop  Safety  Precepts  for  Unmanned  Systems”,  Mr.  Mike  Demmick,  NOSSA 
Track  8 


■  “Defining  Software  Component  Specifications:  An  Open  Approach”,  Mr.  Kenneth  Klein, Computer  Sciences  Corporation 

■  “System  Engineering  and  Software  Exception  Handling  (SEH)”,  Mr.  Herbert  Hecht,  SoHaR  Incorporated 

■  “A  Convergence  of  Technologies  for  Better  Software  NOW!”,  Ms.  Dorothy  Acton,  Lockheed  Martin  IS&GS 

■  “Identifying  Acquisition  Patterns  of  Failure  Using  System  Archetypes”,  Mr.  William  Novak,  Software  Engineering  Institute 

■  “Revitalizing  Education  and  Training  in  Systems  Engineering” ,  Dr.  Don  Gelosh,  Department  of  Defense,  OSD(AT&L)/SSE/ED 

■  “Customer-Driven,  Partnership-Based  Systems  Engineering  Education  and  Training”,  Mr.  Jerrell  Stracener,  Southern  Methodist  University 

■  “Application  of  Systems  Engineering  Principles  in  the  Design  of  Acquisition  Workforce  Curricula”,  (paper)  (slidel  Dr.  David  Olwell.  Naval 
Postgraduate  School 


Thursday.  25  October  2007 
Track  1 


■  “USAF  Type  Certification  of  Commercial  Derivative  Aircraft”,  Mr.  Thomas  Morgan,  USAF 

■  “Global  Positioning  System  Case  Study”,  Mr.  Randall  Bullard,  Air  Force  Center  for  Systems  Engineering 

Track  2 

■  “Implementing  the  Technology  Maturity  Vector”,  Mr.  Joseph  Terlizzese,  Systems  Engineering  Support  Office 

■  “Technology  Readiness  Assessments;  Milestone  B  Certification  Requirement  for  Technologies  to  be  Demonstrated  in  a  Relevant  Environment”, 
Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analyses 

■  “Meeting  Enterprise  System  Engineering  Challenges  for  the  U.S.  Next  Generation  Air  Transportation  System  (NextGen)”,  Mr.  Jerry  Friedman, 
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The  MITRE  Corporation 

■  “Sensor  Resource  Allocation  as  a  Driver  in  System  Concept  Development”,  Mr.  Ravi  Moorthy,  Lockheed  Martin  MS2 
Track  3 

■  “Managing  Requirements  to  Manage  Scope  in  the  Case  of  MUOS”.  Ms.  Christy  Howard.  Maxim  Systems,  Inc. 

■  “Organizational  Leadership  and  Management  Dynamics  for  Technical  Execution  in  Acquisition  Programs”.  Mr.  Francis  Sisti.  Aerospace 
Corporation 

■  “C-17  Systems  Engineering  Process  to  Prioritize  Material  Improvement  Program  (MIP)  Projects”,  Mr.  Thomas  Condron,  USAF  (516 
AESG/ASC) 

■  “How  to  Talk  to  a  Program  Manager”.  Dr.  John  Mishler.  Software  Engineering  Institute 

■  “U.S.  Department  of  Defense  (DoD)  Approach  to  Best  Practices:  Building  Evidence  for  Practice  Selection  Based  on  Real  Experiences”,  Dr. 
Forrest  Shull,  Fraunhofer  Center  Maryland 

Track  4 

■  “Strategic  Focus:  Reduction  of  Total  Ownership  Costs  (R-TOC)  and  Value  Engineering  (VE)”,  Dr.  Danny  Reed,  Institute  for  Defense  Analyses 

■  “Progress  Toward  an  Empirical  Relationships  Between  Reliability  Investments  and  Life-Cycle  Support  Cost”,  Dr.  James  Forbes,  LMI 

■  “Innovation  Strategies  for  Affordable  Readiness”.  Mr.  Tom  Choinski.  Naval  Undersea  Warfare  Center 

■  “Implementing  a  Systems  Engineering  Risk  Program  in  a  Sustainment  Environment”,  Mr.  James  Miller,  USAF 

■  “Asset-Based  PBL  for  Navy  Warships  -  A  case  study  for  LCA  Class  Ships”,  Mike  Mahon 

■  “The  Deployment  Readiness  Service:  The  Challenges  of  Implementing  a  Service  Oriented  Architecture  in  a  Legacy  System  Environment”,  Mr. 
George  Dalton,  USAF 

Track  5 

■  “Aircraft  Flight  Simulator  Acceptance  Criteria”,  Mr.  Dean  Carico,  NAWCAD  PAX 

■  “Computer  Modeling  to  Solve  Problems  with  the  T-38  Propulsion  Modernization  Program”,  Mr.  Randall  Wimer,  USAF  -  FVB 

■  “Efficacy  of  Modeling  &  Simulation  in  Defense  Life  Cycle  Engineering”,  Mr.  Donald  Cox,  Raytheon  Missile  Systems 

■  “Modeling  Integrated  Logistics  Systems  to  Support  Transformation  in  the  U.S.  Coast  Guard”.  Mr.  Patrick  Cumbv.  Vector  CSP,  LLC 

■  “Generic  Sensor  Model”.  Dr.  Stanley  Hack.  Lockheed  Martin  MS-2 

■  “Event  Timeline  Analysis  in  Multi  Mission  Scenarios  with  System  Simulation  Models”,  Mr.  Ravi  Moorthy,  Lockheed  Martin  MS2 
Track  6 

■  “Testing  Concept  of  Operations  in  DoD’s  Net  Centric  Environment”,  Mr.  Steve  Reeder,  South  Carolina  Research  Authority  (SCRA) 

■  “Agile  Governance  for  SOA-Based  Military  Systems  of  Systems”,  Mr.  Robert  Beck,  Villanova  University 

■  “Reducing  Acquisition  Costs  Through  Incremental  Upgrades  by  Migrating  to  SOA”,  Mr.  Tim  Greer,  Lockheed  Martin  Corporation 
Track  7 

■  “The  DoD’s  Proactive  Approach  to  Emerging  Contaminants:  Managing  Risks  Today  for  Tomorrow’s  Warfighter  and  Mission  Readiness”,  Dr. 
Carole  LeBlanc,  Office  of  the  Deputy  Under  Secretary  of  Defense 

■  “ Safe-Escape  Analysis  System  Sa  fety  Engineering  Study”,  (paper!  (slide)  Mr.  David  Hall,  SURVICE  Engineering  Company 

Track  8 


■  “Development  of  Systems  Engineers  in  the  Sensors  &  SONAR  Systems  Department”.  Mr.  Lawrence  Lazar.Naval  Undersea  Warfare 
Center 

■  ^Systems  Engineering  and  the  Art  of  Seeing”,  Dr.  Robert  Monson,  Lockheed  Martin  Corporation 

■  “Understanding  Social  Networks- A  Key  Requirement  for  System  Engineers”,  Mr.  Karl  Selke,  Systems  Engineering  Analyst,  Evidence 
Based  Research,  Inc. 
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Conference  Agenda  {At  A  Glance} 

Sunday,  October  21, 2007 

5:00  pm  -  7:00  pm  Registration  for  Tutorials  and  General  Conference 

(Tutorials  are  an  additional  $225.00  registration  fee) 

Monday,  October  22, 2007 


7:00  am  -  6:00  pm 
7:00  am  -  8:00  am 

8:00  am  - 11:45  am 

12:00  pm  - 1:00  pm 
1:00  pm  -  5:00  pm 
5:00  pm  -  6:00  pm 


Registration 

Continental  Breakfast  for  Tutorial  Attendees  ONLY 
(Tutorials  are  an  additional  $225.00  registration  fee) 

Tutorial  Tracks 

(Please  refer  to  pages  4-5  for  Tutorial  schedule) 

Lunch  for  Tutorial  Attendees  ONLY 
Tutorial  Tracks  Continued 

Reception  in  the  Regatta  Pavilion  (Open  to  All  Participants) 


Tuesday,  October  23, 2007 


7:15  am -6:30  pm 
7:15  am  -  8:15  am 


Registration 
Continental  Breakfast 


8:15  am  -  8:30  am 


8:30  am  -  9:45  am 


9:45  am  - 10:00  am 


Introductions  8C  Opening  Remarks: 

Mr.  Sam  Campagna,  Director ;  Operations ,  NDIA 
Mr.  Boh  Rassa,  Director ;  Systems  Supportability,  Raytheon , 

Chain  Systems  Engineering  Division,  NDIA 

Keynote  Addresses: 

HON James  Finley ,  Deputy  Under  Secretary  of  Defense,  Acquisiton  &  Technology 
HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation 


Break 


2  10th  Annual  Systems  Engineering  Conference  ♦  October  22-25,  2007  ♦  San  Diego,  CA 


10:00  am  - 12:00  pm 

Plenary  Session:  Executive  Panel 

Moderator: 

Mn  Mark  Schaeffer ;  Director ;  Office  of  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics;  Director ;  Systems  and  Software  Engineering 

Panelists: 

Mn  Terry  Jaggers,  SAF/AQR  -  Science,  Technology,  and  Engineering 

Mn  Carl  Siel,  Chief  Engineer,  Office  of  the  Assistant  Secretary  of  the  Navy 
Research,  Development  and  Acquisition 

Mn  Doug  Wiltsie,  HQDA ,  OASA  (ALT) 

12:00  pm  - 1:30  pm 

Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Mn  Mike  Kern,  Senior  Systems  Engineer,  OASD  (Nil) 

1:30  pm  -  5:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

5:00  pm  -  6:30  pm 

Reception  in  the  Regatta  Pavilion 

Wednesday,  October  24, 2007 

7:00  am  -  5:00  pm  Registration 


7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

12:00  pm  - 1:30  pm 

Awards  Luncheon  in  the  Regatta  Pavilion 

1:30  pm  -  5:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

Thursday,  October  25, 2007 

7:00  am  -  3:00  pm  Registration 


7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

12:00  pm  - 1:00  pm 

Luncheon  in  the  Regatta  Pavilion 

1:00  pm  -  3:00  pm 

Concurrent  Sessions 

(Please  refer  to  pages  6-10  for  session  schedule) 

3:00  pm 

Conference  Adjourns 

7:00  am  Registration  &  Continental  Breakfast 
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Tutorial  Sessions  -  Monday,  October  22, 2007 

8:00  am  -  9:45  am  10:15  am  - 1 1 :45  am 


Track  1 

Tutorial 

Session  1A1 

5305  -  Are  we  Ready  for  CMMI®?  If  not. 
Lets  Fix  Ourselves 

Mr.  Al  Florence ,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1A2 

5454  -  Cost  As  an  Independent  Variable 
and  Trade  Studies 

Mr,  Ed  Casy,  Raytheon  Missile  Systems 

Track  3 

Tutorial 

Session  1A3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices 

Mr.  Gary  Langford,  The  Naval  Postgraduate 

School 

Track  4 

Tutorial 

Session  1A4 

5498  -  System  Verification  Organization 

Mr,  Jeffrey  Grady,  JOG  System  Engineering, 
Inc. 

Track  5 

Tutorial 

Session  1A5 

Track  6 

Tutorial 

5542  -  Best-In-Class  Early  Defect 

Detection  and  Defect  Prevention 

Session  1A6 

Mr.  Timothy  Olson,  Lean  Solutions  Institute, 
Inc. 

Track  7 

Tutorial 

5784  -  Operational  Concepts:  Using  Cases 
&  Scenarios  to  Understand  Users  Needs 

Session  1A7 

Mr.  Raymond  Jorgensen,  Rockwell  Collins 

Track  8 

Tutorial 

Session  1A8 

5577  -  Introduction  to  SysML  & 

Object  Oriented  Systems  Engineering 
Methodology  (OOSEM) 

Mr.  Ahe  Meilich,  Lockheed  Martin 

Track  1 

Tutorial 

Session  1B1 

5305  -  Are  we  Ready  for  CMMI®?  If  not. 
Lets  Fix  Ourselves  (Cont'd) 

Mr.  Al  Florence,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1B2 

5454  -  Cost  As  an  Independent  Variable  and 
Trade  Studies  (Contd) 

Mr.  Ed  Casy,  Raytheon  Missile  Systems 

Track  3 

Tutorial 

Session  1B3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices  (Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate 
School 

Track  4 

Tutorial 

Session  1B4 

5498  -  System  Verification  Organization 
(Cont'd) 

Mr.  Jeffrey  Grady,  JOG  System  Engineering, 
Inc. 

Track  5 

Tutorial 

Session  1B5 

Track  6 

Tutorial 

Session  1B6 

5542  -  Best-In-Class  Early  Defect 

Detection  and  Defect  Prevention  (Cont'd) 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc. 

Track  7 

Tutorial 

Session  1B7 

5784  -  Operational  Concepts:  Using  Cases 
&  Scenarios  to  Understand  User's  Needs 
(Cont'd) 

Mr.  Raymond  Jorgensen,  Rockwell  Collins 

Track  8 

Tutorial 

Session  1B8 

5577  -  Introduction  to  SysML  & 

Object  Oriented  Systems  Engineering 
Methodology  (OOSEM)  (Cont'd) 

Mr.  Ahe  Meilich,  Lockheed  Martin 
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Buffet  Lunch 


3:15  pm  -  5:00  pm 


1 :00  pm  -  2:45  pm 


Track  1 

Tutorial 

Session  1C1 

5307  -  Requirements  Development  and 
Management 

Mr.  Al  Florence ,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1C2 

5326  -  Integrating  Systems  Engineering 
with  Earned  Value  Management 

Mr.  Paul  Solomon,  Performance-Based  Earned 
Value • 

Track  3 

Tutorial 

Session  1C3 

5404  -  How  to  Reduce  Schedule  Uncertainty 
by  Integrating  Sound  Management  Methods 
with  Systems  Engineering  Best  Practices 
(Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate  School 

Track  4 

Tutorial 

Session  1C4 

5511  -  Modeling  Sustainment  and  Risk 
Mitigation  for  Net-Enabled  Realities 

Mr.  Philip  Boxer,  Software  Engineering 
Institute/CMU 

Track  5 

Tutorial 

Session  1C5 

5540  -  Introduction  to  Reliability  Analysis 

Dr.  Meng-Lai  Yin,  Raytheon  Company 

Track  6 

Tutorial 

Session  1C6 

5544  -  How  to  Define  Practical  Systems 
Engineering  Metrics 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc. 

Track  7 

Tutorial 

Session  1C7 

5582  -  Leading  Effective  System  of 

Systems  (SoS)  Technical  Reviews 

Mr.  David  Walden,  Sysnovation,  LLC 

Track  8 

Tutorial 

Session  1C8 

5577  -  Introduction  to  SysML  &  Object 
Oriented  Systems  Engineering 

Methodology  (OOSEM) 

Dr.  Ahe  Meilich,  Lockheed  Martin 

Track  1 

Tutorial 

Session  1D1 

5307  -  Requirements  Development  and 
Management  (Cont'd) 

Mr.  Al  Florence,  The  MITRE  Corporation 

Track  2 

Tutorial 

Session  1D2 

5326  -  Integrating  Systems  Engineering 
with  Earned  Value  Management  (Cont'd) 

Mr.  Paul  Solomon,  Performance-Based  Earned 
Value • 

Track  3 

Tutorial 

Session  1D3 

5404  -  How  to  Reduce  Schedule  Uncertainty  by 
Integrating  Sound  Management  Methods  with 
Systems  Engineering  Best  Practices  (Contd) 

Mr.  Gary  Langford,  The  Naval  Postgraduate  School 

Track  4 

Tutorial 

Session  1D4 

5511  -  Modeling  Sustainment  and  Risk 
Mitigation  for  Net-Enabled  Realities 
(Cont'd) 

Mr.  Philip  Boxer,  Software  Engineering 
Institute/CMU 

Track  5 

Tutorial 

Session  1D5 

5540  -  Introduction  to  Reliability  Analysis 
(Cont'd) 

Dr.  Meng-Lai  Yin,  Raytheon  Company 

Track  6 

Tutorial 

Session  1D6 

5544  -  How  to  Define  Practical  Systems 
Engineering  Metrics 

Mr.  Timothy  Olson,  Lean  Solutions 

Institute,  Inc . 

Track  7 

Tutorial 

Session  1D7 

5582  -  Leading  Effective  System  of 

Systems  (SoS)  Technical  Reviews  (Cont'd) 

Mr.  David  Walden,  Sysnovation,  LLC 

Track  8 

Tutorial 

Session  1D8 

5577  -  Introduction  to  SysML  &  Object 
Oriented  Systems  Engineering 

Methodology  (OOSEM)  (Cont'd) 

Dr.  Ahe  Meilich,  Lockheed  Martin 

5:00  pm  -  6:00  pm  Reception  in  Regatta  Pavilion 


Tuesday,  October  23,  2007 


5:00  pm  -  6:00  pm  Reception  in  the  Regatta  Pavilion 


Wednesday,  October  24,  2007 


12:00  pm  Lunch  in  the  Regatta  Pavilion 


Wednesday,  October  24,  2007 
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Thursday,  October  25,  2007 

8:00  am  -  9:45  am  concurrent  sessions  10:15  am  - 12:00  pm 
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12:00  pm  Lunch  in  the  Regatta  Pavilion 


Thursday,  October  25,  2007 


concurrent  sessions 

:30  pm  -  3:00  pm 

Bayview 

A 

TRACK  2 

Systems  Engineering 
Effectiveness 

Session  4C2 

5814  -  Defense  System  and 
Large-Scale  Systems  Based  on 
Terminal  Control 

Mr.  Xiang-Wen  Xiong,  Zhongheng 
High-Tech  Institute,  Inc. 

Bayview 

B 

TRACK  4 

Logistics,  Supportability, 
and  Readiness 

Session  4C4 

5416  -  The  Deployment  Readiness 
Service:  The  Challenges  of 
Implementing  a  Service  Oriented 
Architecture  in  a  Legacy  System 
Environment 

Mr.  George  Dalton,  USAF 

yview 

C 

TRACK  5 

Modeling  & 

Simulation 

5838  -  Generic  Sensor  Model 

5852  -  Event  Timeline  Analysis 
in  Multi  Mission  Scenarios  with 
System  Simulation  Models 

5428  -  A  Prototype  Tool  for 
Concept  Design  Modeling  and 
Optimization  of  Combat  Systems 

CQ 

Session  4C5 

Dr.  Stanley  Hack,  Lockheed 

Martin  MS-2 

Mr.  Ravi  Moorthy,  Lockheed 
Martin  MS2 

Mr.  Vikram  Ganesan,  General 
Dynamics  Land  Systems 

Conference  Adjourns 


Systems  Engineering  Effectiveness: 

Mr.  A1  Brown 
Ms.  Sharon  Vannucci 
Mr.  Bob  Lyons 

Logistics  Supportability  &  Readiness: 

Mr.  Chuck  Silva 
Mr.  Joel  Moorvich 

Test  &  Evaluation  in  Systems  Engineering: 
Col  Rich  Stuckey,  USAF 
Mr.  Tom  Wissink 
Program  Management: 

Mr.  Hal  Wilson 
Modeling  &  Simulation: 

Mr.  Jim  Hollenbach 
Mr.  Gary  Belie 
Net  Centric  Operations: 

Mr.  Jack  Zavin 
Dr.  Rich  Eilers 
Dr.  Tom  Wickstrom 
Best  Practices  &  Standardization: 

Mr.  Paul  Croll 
Software: 

Dr.  Tom  Christian 
Education  &  Training: 

Mr.  George  Mooney 
Integrated  Diognostics: 

Mr.  Howard  Savage 
Mr.  Dennis  Hecht 
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1A4 
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Mr.  Jeffrey  Grady 

1A6 
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1A7 
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1C1 
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Maj  Joerg  Walter,  US AF 

Dr.  Som  Soni 

2C4 
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Environment,  Safety,  and  Occupational  Health  (ESOH)  -  Design 
Considerations  to  Support  Sustainability  and  Readiness 

Ms.  Patricia  Huheey 

Ms.  Karen  Gill 

2C5 

5421 

M&S -Related  Guidelines  for  Contracting 

Mr.  Michael  Truelove 

2C5 
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Implementing  the  Acquisition  M&S  Master  Plan 
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•  CMMI-SVC  News 

•  Overview  of  the  draft  CMMI  for  Services  (CMMI-SVC) 

•  What  is  the  CMMI? 

•  Why  is  the  CMMI-SVC  needed? 

•  How  are  services  different? 

•  What  is  the  basis  for  the  CMMI-SVC  model? 

•  What  is  the  scope  and  content  of  the  CMMI-SVC? 

•  Feedback  to  date 

•  What  was  the  result  of  the  expert  review? 

•  What  was  the  experience  of  the  pilot  projects? 

•  Next  Steps 

•  What  is  the  schedule? 

•  How  can  I  participate? 
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CMMI  Steering  Group  to 
Address  CMMI  for  Services 


CMMI 


•  There  was  a  serious  concern  that  concurrent 
development  of  the  CMMI-ACQ  and  CMMI-SVC 
models  would  stress  the  SEI  resources  needed  to 
deliver  the  CMMI-ACQ  model  on  time.  Now  that 
CMMI-ACQ  is  almost  released,  the  SEI  resources 
are  available  to  go  forward  with  the  CMMI-SVC. 

•  The  CMMI-SVC  team  will  address  past  Steering 
Group  concerns  at  their  Nov  meeting  and  present 
a  plan  to  complete  development. 


What  is  a  Capability  Maturity 
Model  (CMM)? 


CMMI 


•  A  conceptual  framework  for  structuring,  understanding,  and 
evaluating  the  capability  and  maturity  of  an  organization’s 
processes 

•  more  than  a  laundry  list  of  best  practices 

•  more  than  a  collection  of  benchmarks  and  metrics 

•  A  tool  that  enables  meaningful,  in-depth  organizational 
assessment 

•  internally 

•  externally 

•  A  map  that  guides  practical  process  improvement  and 
institutionalizes  it 

•  How  to  you  get  from  here  to  there  and  stay  there ? 


What  is  the  CMMI? 


CMMI 


The  CMM  Integration  (CMMI)  of  multiple  CMMs  into  a 
single  unified  framework 


EIA  Interim  Standard  731 , 
System  Engineering 
Capability  Model  (SECM) 


Capability  Maturity 
Model  for  Software  V2, 
draft  C  (SW-CMM  V2C) 


Integrated  Product 
Development 
Capability  Maturity 
Model,  draft  V0.98 
(IPD-CMM) 


Software  Acquisition 
Capability  Maturity  Model, 
version  1 .01  (SA-CMM) 


SEI 


Government 


■  CMMI  1 

|  Product  Suite  | 

Three  complementary 
constellations 


CMMI 


<e> 


CMMI-DEV 

CMMI-SVC 

provides  guidance 

provides  guidance  for 

for  measuring, 

those  providing 

monitoring,  and 

services  within 

managing 

organizations  and  to 

development 

external  customers 

processes 

Core 
Model 
Foundation 


CMMI-ACQ 

provides  guidance 
to  enable 
informed  and 
decisive 
acquisition 
leadership 


Courtesy  of  the  SEI 
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Why  is  CMMI  for  Services 
(CMMI-SVC)  needed? 


CMMI 


Customer  discontent 
Service  society 
Legislation 

Government  and  industry 
trends 
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How  are  services  different? 


•  Services  form  a  distinctive  category  of  products 

•  A  service  is  an  intangible,  non-storable  product 

•  What  makes  a  service  intangible  or  non-storable? 

•  Customer  desires  a  situation  or  state  (e.g.,  to  have  high  network 
availability)  rather  than  a  tangible  artifact 

•  Provider  delivers  value  without  allowing  the  customer  independent, 
unrestricted  means  to  generating/employing  that  value  (e.g.,  leasing 
vehicles) 

•  Product  delivery  requires  continuing  application  of  labor  (e.g.,  operation 
of  a  facility) 

•  Services  imply  customer/provider  relationships  governed  by 
service  agreements 

Service  and  non-service  products  may  be  delivered  as  part  of  a  single 

agreement  (e.g.,  training  that  includes  hardcopy  materials) 

•  Services  are  often  delivered  via  the  operation  of  a  service  system 


Service  system 


ft 

CMMI 


•  A  necessary  concept  for  understanding  the  effective 
delivery  of  services 

•  An  integrated  and  interdependent  combination  of 
processes,  resources,  and  people  that  satisfies  service 
requirements. 

•  Portions  are  not  delivered  to  the  customer  or  end-user  as 
part  of  service  delivery 

•  Portions  may  remain  owned  by  the  customer  or  end-user 
before  service  delivery  begins  and  after  service  delivery 
ends. 

•  Encompasses  everything  required  for  service  delivery, 
including  work  products,  processes,  infrastructure, 
consumables,  and  customer  resources. 
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What  is  the  scope  of 
CMMI-SVC? 


CMMI 


•  Covers  practices  required  to  manage,  establish,  and  deliver 
services,  in  four  process  area  categories 

•  Project  (service)  management 

•  Process  management 

•  Service  support 

•  Service  establishment  and  delivery 

•  Intended  to  match  the  scope  of  the  definition  of  services 

•  Broad  applicability  to  a  range  of  service  domains 

•  Information  technology,  engineering,  defense,  transportation, 
finance,  health  care 

•  Staff  augmentation  services  need  careful  consideration 

•  How  do  you  evaluate  process  improvement  for  processes  over 
which  you  have  no  control? 
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CMMI-SVC  Process  Areas 


CMMI 


Process  Management 

Organizational  Innovation  and 
Deployment  (OID) 

Organizational  Process  Definition  (OPD) 
Organizational  Process  Focus  (OPF) 

Organizational  Process  Performance 
(OPP) 

Organizational  Service  Management 
(OSM) 

Organizational  Training  (OT) 

Service  Support 

Causal  Analysis  and  Resolution  (CAR) 
Configuration  Management  (CM) 
Decision  Analysis  and  Resolution  (DAR) 
Measurement  and  Analysis  (MA) 

Problem  Management  (PRM) 

Process  and  Product  Quality  Assurance 
(PPQA) 


Service  Establishment  and  Delivery 

•  Incident  and  Request  Management 
(IRM) 

•  Service  Delivery  (SD) 

•  Service  System  Development  (SSD) 

•  Service  Transition  (ST) 

Project  Management 

•  Capacity  and  A  vail  ability  Management 
(CAM) 

•  Integrated  Project  Management  (IPM) 

•  Project  Monitoring  and  Control  (PMC) 

•  Project  Planning  (PP) 

•  Requirements  Management  (REQM) 

•  Risk  Management  (RSKM) 

•  Quantitative  Project  Management  (QPM) 

•  Service  Continuity  (SCON) 

•  Supplier  Agreement  Management  (SAM) 
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Services-specific  PAs 


ft 

CMMI 


Process  Area 

Maturity  Level 

Specific  Goals/ 
Practices 

Capability  and  Availability  Management  (CAM) 

3 

2/6 

Incident  and  Request  Management  (IRM) 

2 

2/6 

Organizational  Service  Management  (OSM)* 

3 

2/7 

Problem  Management  (PRM) 

3 

2/7 

Service  Continuity  (SCON)* 

3 

3/10 

Service  Delivery  (SD) 

3 

2/7 

Service  System  Development  (SSD)  * 

3 

3/12 

Service  Transition  (ST) 

3 

3/12 

*  optional  process  areas  (independent  named  additions) 
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CMMI-SVC  Level  2  PAs 


ft 

CMMI 


•  Incident  and  Request  Management 

•  To  ensure  the  timely  resolution  of  requests  for  service 
and  incidents  that  occur  during  service  delivery 

•  Requirements  Management 

•  Extended  from  the  Core  Model  Foundation  with  an 
additional  goal 

•  To  include  the  establishment  and  maintenance  of  written 
agreements  between  service  providers  and  customers 
on  service  requirements  and  service  levels. 

•  Six  other  level  2  PAs  from  the  CMF 
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CMMI-SVC  Level  3  PAs 
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CMMI 


•  Capacity  and  Availability  Management 

•  To  plan  and  monitor  the  effective  provision  of  resources 
to  support  service  requirements 

•  Problem  Management 

•  To  prevent  incidents  from  recurring  by  identifying  and 
addressing  underlying  causes  of  incidents 

•  Service  Delivery 

•  To  deliver  services  in  accordance  with  service 
agreements 

•  Service  Transition 

•  To  deploy  new  or  significantly  changed  service  systems 
while  managing  their  effect  on  ongoing  service  delivery 
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Optional  PAs  for  CMMI-SVC 
Level  3 


^  • 

CMMI 


•  Organizational  Service  Management 

•  To  establish  and  maintain  standard  services  that  ensure 
the  satisfaction  of  the  organization's  customer  base 

•  Service  Continuity  Management 

•  To  establish  and  maintain  contingency  plans  for 
continuity  of  agreed  services  during  and  following  any 
significant  disruption  of  normal  operations 

•  Service  System  Development 

•  To  analyze,  design,  develop,  integrate,  and  test  service 
systems  to  satisfy  existing  or  anticipated  service 
agreements 


14 


What  was  the  result  of  the 
expert  review? 


^  • 

CMMI 


•  An  expert  review  was  held  Jan  23  -  Mar  23,  2007 

•  500+  reviewers,  representing: 

•  50  companies, 

•  14  DoD  organizations, 

•  4  academic  institutions,  and 

•  7  professional,  governmental,  or  research  centers 

•  Reviewers  included  SEI  transition  partners 

•  Response  showed  strong  interest  in  CMMI-SVC 

•  900+  change  requests  compares  favorably  to  those 
received  for  CMMI-DEV 

•  50  survey  responses  to  architectural  questions 
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What  was  the  result  of  the 
expert  review?  (more) 


^ — 

CMMI 


<g) 


•  Reviews  commented  mostly  on  CMM-SVC  architecture  &  Common 
Model  Foundation  material 

•  CRs  were  distributed  equally  among  categories  related  to  SVC  PAs 

•  CMMI-SVC  team  has  analyzed  all  architectural  CRs;  most  have  a 
proposed  resolution 

•  CRs  showed  excellent  depth  of  insight  and  rich  informative  content 
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Sample  Survey  Responses 


•  The  service  practices  that  are  covered  in  CMMI-SVC  will  enable  service  organizations  to  provide  more 
effective  support  to  their  customers. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

78.9% 

8.8% 

12.3% 

•  The  material  in  CMMI-SVC  yields  a  useful  adaptation  of  CMMI  best  practices  as  they  relate  to  service 
deployment. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

66.7% 

14.0% 

15.8% 

•  CMMI-SVC  does  not  impose  constraints  (derived  from  the  needs  of  a  specific  service  or  market 

segment)  that  would  limit  or  prevent  other  organizations  from  adapting  the  model  to  their  own  specific 
needs. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

55.6% 

29.6% 

27.8% 

•  The  CMMI-SVC  is  easy  to  understand  and  apply  to  a  service  organization. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

42.8% 

27.8% 

29.6% 
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What  was  the  experience  of 
the  pilot  projects? 


CMMI 


•  Planned  pilots  were  postponed 

•  CMMI-SVC  participating  companies  piloted  the  model  internally 

•  Characteristics  of  the  piloted  organizations: 

•  Most  had  implemented  CMMI-DEV 

Some  had  separate  ITIL  and  ISO  20000  initiatives 
Most  are  moving  towards  integration  under  CMMI  umbrella 

•  The  pilots  represented  the  following  service  domains: 


Company 

Service  Domains 

SSCI 

IT  Application  Operations  &  Support 

DNV-CIBIT 

Banking 

Northrop  Grumman 

Logistics,  HR,  IT,  Applications  O&M 
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What  did  the  pilots  see  as 
benefits? 


^  • 

CMMI 


•  Improved  quality  of  services 

•  Encouraged  a  disciplined  culture  for  service  management 

Better  management  visibility  into  services 
Fewer  surprises 

•  Fosters  process  improvement 

•  Less  Interpretation  issues  (&  appraisal  expense)  than  with  CMMI-DEV 

•  Applying  a  CMMI  process  to  the  services  brought  credibility  and  buy-in 
from  stakeholders 


•  Increased  sharing  between  development  and  services  communities 
Common  processes 
Standard  terminology 

Integrated  process  improvement  standards  and  models 


Encouraged  end-to-end  lifecycle  process  approach  helping  to  identify 
service  requirements,  ease  deployment  issues,  reduce  stove-piped 
groups,  and  improve  efficiencies  of  support-related  groups  (IT 
Applications) 
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What  did  the  pilots  see  as 
challenges? 
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CMMI 


•  Obtaining  funding  in  environments  that  are  primarily  LOE-based 

•  Differences  in  terminology  between  development  and  services 

Terms  like  “Project”  (funding  period),  “Product”  (service),  “Work  Product”, 
“Product  Component”,  “Requirement” 

Interpreting  CMMI’s  “project”  term  for  services 

•  No  standard  life-cycle  definition  for  services 

•  Instilling  project  management  culture  in  services 

Weak  in  using  requirements  for  planning  and  negotiating  resources  and 
activities 

•  Ownership  of  service  system  components  not  as  clear 

•  Release  management  and  deployment  to  non-standardized,  constantly 
changing  environments 

•  Finding  CMMI-knowledgeable  individuals  who  also  know  services 

•  Integrating  process  groups  and  assets 

•  Services  where  customer  and  provider  share  resources  and  processes 

•  Staff  augmentation 


20 


<e> 

CMMI 

Issues  to  Address 


•  What  is  the  business  case  for  the  CMMI-SVC? 

•  What  distinguishes  CMMI-SVC  from  CMMI-DEV  (vl  .2)  and 
other  models? 

•  What  are  the  characteristics  of  service  providers  and  how 
are  they  represented  in  the  CMMI-SVC? 

•  Can  the  broad  spectrum  of  services  be  governed  by  a 
single  model? 

•  How  will  the  Services  Sector  be  engaged? 

•  What  are  the  impacts  to  small  businesses? 

•  How  will  CMMI-SVC  be  used  with  other  CMMI  products? 
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What  is  the  schedule? 


CMMI 


•  CMMI-SVC  team  will  meet  to  review  additional 
requirements  and  re-plan  remaining  work  (early  Nov) 

•  Detailed  schedule  is  pending 

•  A  preliminary  estimate  for  release  of  CMMI-SVC,  vl  .2  is 
4th  quarter  2008 


ID 

Task  Name 

2005 

20O6 

20O7 

20O8 

Qtr  1  Qtr  2  Qtr  3  Qtr  4 

Qtr  1  Qtr  2  Qtr  3  Qtr  4 

Qtr  1  Qtr  2  Qtr  3 

Qtr  4 

Qtr  1  Qtr  2  Qtr  3  Qtr  4 

4 

CMMI-SVC,  vO.5 

5 

CMMI-SVC,  vO.5  review 

6 

CMMI-SVC,  vl  .2 
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How  can  I  participate? 


CMMI 


•  Get  more  information  about  CMMI-SVC 

•  CMMI  web  page  -  http://www.sei.cmu.edu/cmmi/ 

•  CMMI  for  Services  Public  Workspace 

(http://bscw.sei.cmu.edU/bscw/bscw.cai/0/424939)  contains: 

•  Draft  CMMI-SVC  model,  v0.5 

•  Information  on  joining  CMMI-SVC  information  email  list 

•  Review  draft  CMMI-SVC  release 

•  If  already  experienced  in  CMMI,  consider  piloting  the  model 

•  Other  opportunities  may  exist  as  a  result  of  the  CMMI-SVC 
re-planning  effort;  watch  CMMI-SVC  public  workspace  for 
updates 
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Backup 


CMMI 
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Who  is  working  on 
CMMI-SVC? 


•  Development  Team 

Craig  Hollenbach  (Northrop  Grumman) 
Roy  Porter  (Northrop  Grumman) 
Brandon  Buteau  (Northrop  Grumman) 
Lynn  Penn  (Lockheed  Martin) 

•  Frank  Niessink  (DNV/CIBIT) 

Jerry  Simpson  (SAIC) 

•  Drew  Allison  (SSCI) 

Eileen  Forrester  (SEI) 

Barbara  Tyson  (SEI) 

•  Eileen  Clark  (SRA) 

•  Other  contributors 

Jeff  Zeidler  (Boeing) 

•  Rich  Raphael  (Mitre) 

Joanne  O’Leary  (SEI) 


CMMI 
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CMMf 

General  Survey  Questions 


1.  The  service  practices  that  are  covered  in  CMMI-SVC  will  enable  service  organizations  to 
provide  more  effective  support  to  their  customers. 

2.  The  material  in  CMMI-SVC  yields  a  useful  adaptation  of  CMMI  best  practices  as  they 
relate  to  service  deployment. 

3.  The  CMMI-SVC  appropriately  uses  the  CMMI  framework. 

4.  CMMI-SVC  includes  process  areas  that  must  be  satisfied  for  process  improvement  and 
institutionalization. 

5.  CMMI-SVC  does  not  impose  constraints  (derived  from  the  needs  of  a  specific  service  or 
market  segment)  that  would  limit  or  prevent  other  organizations  from  adapting  the  model 
to  their  own  specific  needs. 

6.  The  CMMI-SVC  is  easy  to  understand  and  apply  to  a  service  organization. 

7.  The  process  areas  in  CMMI-SVC  cover  all  significant  service-specific  requirements  and 
effectively  reflect  activities  that  a  service  organization  should  be  accomplishing. 

8.  Additions  and  amplifications  that  exist  in  other  models  and  are  also  used  within  the  CMMI- 
SVC  constellation  are  appropriate. 

9.  Notes  and  examples  in  CMMI-SVC  clearly  apply  to  service  organizations  and  meet  their 
specific  needs. 

10.  References  in  PAs  to  related  process  areas  are  clear  and  consistently  applied. 
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Results  to  General  Survey 


ft 

CMMI 
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Process  Area  Questions 


CMMI 


a.  Problem  management  practices  that  are  common  within  the  service  industry  are  appropriately 
addressed  in  the  process  area  Problem  Management  and  are  distinguished  from  the  practices  in  the 
Causal  Analysis  and  Resolution  process  area. 

b.  The  Project  Management  category  is  the  most  appropriate  classification  for  the  Service  Continuity 
Management  and  Capacity  and  Availability  Management  process  areas. 

c.  The  Process  Management  category  is  the  most  appropriate  classification  for  the  Organizational 
Service  Management  process  area 

d.  The  practices  within  the  Service  Continuity  process  area  should  build  upon  the  practices  within  the 
Risk  Management  process  area  similar  to  the  manner  in  which  the  Integrated  Project  Management 
process  area  builds  upon  maturity  level  2  project  management  practices. 

e.  The  Service  System  Development  process  area  must  be  required  for  an  organization  to  be  a  mature 
service  organization. 

f.  The  specific  practices  in  the  Service  System  Development  process  areas  are  presented  with  the 
appropriate  rigor  and  detail  for  a  mature  service  organization. 

g.  The  Project  Monitoring  and  Control  process  area  adequately  addresses  service  level  management. 

h.  Material  about  the  collection  of  customer  satisfaction  information  is  adequately  covered  as  a  specific 
practice  in  Organizational  Service  Management  (an  optional  process  area)  and  as  informative  material 
in  the  Service  Delivery  process  area. 

i.  Maintenance  found  in  the  Service  Delivery  process  area  is  adequately  differentiated  from  product 
maintenance  covered  by  CMMI-DEV. 

j.  The  IPPD  addition  is  as  appropriate  or  as  applicable  for  CMMI-SVC  as  it  is  for  CMMI-DEV  and  should 
be  added. 

k.  The  Supplier  Agreement  Management  process  area  is  appropriate  both  for  organizations  with  tangible 
products  and  service  organizations  with  supplier  agreements  solely  for  services. 

l  The  Supplier  Agreement  Management  process  area  should  be  required  to  reach  maturity  level  2  for 
service  organizations  with  supplier  agreements  solely  for  services  (as  it  is  for  organizations  with 
suppliers  of  tangible  products). 
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Process  Area  Survey  Questions 


CMMI 


<e> 


90.0% 


80.0% 


70.0% 


Process  Area  Survey  Questions 


V'  .O  /</  S  .O  ^  c,x  ^  ^  „_v 


a 


$&&&&&&  &  &  &  &  & 


□  Agree  or  Strongly  Agree 

□  Neutral 

n  Disagree  or  Strongly 
Disagree 
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What  is  the  relationship 
between  CMMI-SVC  and  ITIL? 


CMMI 


•  CMMI-SVC  complements  ITIL 

•  Summarizes  ITIL  best  practices  into  a  small  set  of 
specific  practices. 

•  Reuses  about  80%  of  the  current  CMMI  model,  allowing 
users  to  leverage  their  investments  in  development- 
based  process  training,  improvements,  and  infrastructure 
to  serv  ce-based  offerings. 

•  Provides  an  industry-accepted  maturity  model,  helping 
organizations  to  plan  and  track  their  incremental 
progress  toward  high  maturity. 

•  Uses  the  same  SCAMPI  appraisal  method  that  is  used 
with  the  current  CMMI  model,  allowing  organizations  to 
leverage  appraisal  expertise,  preparation  methods,  and 
selected  artifacts. 
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Who  uses  CMMs? 
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Commercial/In-house 


67.6% 


Contractor  for 
Military /Government 


28.8% 


Military /Government 
Agency 


0  200  400  600  800  1000 

Number  of  Organizations 


1200 


Courtesy  of  the  SEI 
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Why  do  CMMs  really  matter? 


CMMI 


Improvements 

Median 

Data 

Count 

Low 

High 

Cost 

34% 

29 

3% 

87% 

Schedule 

50% 

22 

2% 

95% 

Productivity 

61% 

20 

11% 

329% 

Quality 

48% 

34 

2% 

1 32% 

Customer 

Satisfaction 

14% 

7 

-4% 

55% 

ROI 

4.0  :  1 

22 

1.7  :  1 

27.7  :  1 

•  N  =  30,  as  of  August  2006 

•  Organizations  with  results  expressed  as  change  over  time 


Courtesy  of  the  SEI 
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Revitalizing 

Education  and  Training  in 
Systems  Engineering 


Don  S.  Gelosh,  PhD 

Sr.  Systems  Engineer 

Office  of  Deputy  Director  for  Enterprise  Development 
Systems  and  Software  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense  (A&T) 


Outline 


>  Mission  Statement 

>  Systems  Engineering  Policies 

>  Education  and  Training 

>  Systems  Planning,  Research, 
Development,  and 
Engineering  (SPRDE) 

Career  Field  Update 

>  Core  Plus 

>  The  Way  Ahead... 


Systems  and  Software  Engineering 

Mission  Statement 


>  Shape  acquisition  solutions  and  promote  early  technical  planning 

>  Promote  the  application  of  sound  systems  and  software  engineering, 
developmental  test  and  evaluation,  and  related  technical  disciplines 
across  the  Department's  acquisition  community  and  programs 

>  Raise  awareness  of  the  importance  of  effective  systems  engineering 
and  drive  the  state-of-the-practice  into  program  planning  and 
execution 

>  Establish  policy,  guidance,  best  practices,  education,  and  training  in 
collaboration  with  academia,  industry,  and  government  communities 

>  Provide  technical  insight  to  program  managers  and  leadership  to 
support  decision  making 


Driving  Technical  Excellence  into  Programs! 
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System  Engineering  Policies 


All  programs  shall  develop  a  SE 
Plan  (SEP) 

Each  PEO  shall  have  a  lead  or  chief 
systems  engineer  who  monitors  SE 
implementation  within  program 
portfolio 

Event-driven  technical  reviews  with 
entry  criteria  and  independent 
subject  matter  expert  participation 

OSD  shall  review  program’s  SEP  for 
major  acquisition  programs  (ACAT 
ID  and  1AM) 


Technical 

Planning 


Technical 

Leadership 


Technical 

Execution 

J 


Technical 

Excellence 


Strong  technical  foundation  is  the  value  of 
systems  engineering  to  the  program  manager 
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What’s  Coming  in  Policy 


>  Codified  SE  revitalization  in  DoDI  5000.2 

•  Captures  previously  approved  SE  and  related  policies 

•  Mandates  SEP  at  Milestones  A,  B,  and  C 

•  Considers  SE  during  Concept  Refinement  and  Technology 
Demonstration  phases 

•  Mandates  system-level  Critical  Design  Review,  sets  CDR  exit  criteria, 
requires  a  CDR  report  to  Milestone  Decision  Authority 

•  Establishes  functional,  allocated,  and  product  baselines  during  SDD 

•  Mandates  Program  Support  Reviews  for  all  MDAPs 

•  Establishes  requirement  for  Configuration  Management  and  Data 
Management  strategies 
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SE  in  Defense  Acquisition  Guidebook 


>  SE  guidance  to  acquisition  community — Chapter  4 

•  Best  practices  for  “applied”  SE 

-  SE  processes  (8  technical  management,  8  technical) 

-  Guide  for  each  acquisition  phase,  concept  refinement 
through  disposal 

•  Linkage  of  SE  products  and  processes  to 
acquisition  objectives  and  decision  points 

•  Currently  being  updated 


http://akss.dau.mil/daa/welcome.asp 
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Education  and 
Training 


Education  &Training  Background 


>  In  October  2003,  an  Education  and  Training  Summit  found  that 
while  SE  processes  were  sound,  their  application  in 
acquisition  programs  was  often  lacking  in  rigor. 

>  Among  other  initiatives,  the  Director,  DS/SE,  (now  SSE/ED) 
issued  a  directive  to  re-baseline  the  SE  competencies  and 
curriculum  for  the  SPRDE/SE  career  path. 

>  The  SPRDE/SE  FIPT,  working  with  the  Institute  for  Defense 
Analysis,  developed  almost  200  learning  outcomes  to  serve  as 
a  basis  for  the  new  curriculum. 

>  The  new  curriculum  was  structured  to  focus  on  the  16  DoD  SE 
processes  at  Level  1, 5  SE  phases  at  Level  II,  and  leadership 
and  management  skills  at  Level  III. 

>  From  August  2004  until  February  2007,  DAU  developed  four 
new  courses:  SYS  101 ,  SYS  202,  SYS  203,  and  SYS  302. 
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Education  &  Training  (SYS  101) 


>SYS  101:  Fundamentals  of  Systems  Planning,  Research, 
Development  and  Engineering 


•  Technically  rigorous,  comprehensive  online  course  that 
provides  an  introduction  to  systems  engineering. 

•  Based  around  the  8  technical  management  processes  and 
the  8  technical  processes  outlined  in  the  Defense  Acquisition 
Guidebook. 


Also  suitable  for  personnel  in  technical  management  and 
program  management  positions  who  want  to 
understand  more  about  systems 
engineering  and  the  details  of  its  J1 

processes. 
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Education  &  Training  (SYS  202) 


>  SYS  202:  Intermediate  Systems  Planning,  Research, 
Development  and  Engineering,  Part  I 

•  Intermediate-level  online  course  that  provides  a  description 
of  how  the  SE  processes  can  be  applied  within  the  context  of 
the  various  phases  of  the  Defense  acquisition  framework. 

*  Course  content  includes  the  scope  and  role  of  SE  and  its  key 
technical  inputs  and  outputs; 


the  key  aspects  of  technical  baselines 
and  the  role  of  technical  reviews; 
and  important  design  considerations. 
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Education  &  Training  (SYS  203) 


>SYS  203:  Intermediate  Systems  Planning,  Research,  Development 
and  Engineering,  Part  II 

•  Intermediate-level  1-week  long  classroom  course  that  requires 
students  to  apply  the  DoD  Systems  Engineering  processes  and 
techniques  learned  in  SYS  101  &  SYS  202. 


Students  work  in  integrated  product  teams  and  apply  systems 
engineering  technical  processes  and  technical  management 
processes  to  a  defense  system 
across  the  various  phases 
of  the  Defense  acquisition 
framework. 


Systems  Engineering  Technical  Review  Timing 


Work 

Efforts 


Concept 

Refinement 


<^>  Concept  Decision 


Technology 

Development 


System  Development  & 
Demonstration 


System  Demonstration 


Production  & 

Operations  & 

Deployment 

Support 

LRIP/IOT&E/Full  Rate 

Sustainment 

Production  &  Deployment 

FRP  Decision  Review  ^ 

y  Disposal 

Technical 

Baseline 


A 

ASR 


Preferred 

System 

Concept 


TRA 


A  A  A 

SRR  IBR  SFR 


A 

PDR 


A 

CDR 


TRA 


A  A  A 

TRR  FRR  SVR/FCA/ 
PRR 


A  A 

OTRR  PCA 


System  System  Allocated  Product 
Specification  Functional  Baseline  Baseline 
Baseline 


▼  Technology  Readiness  Assessment 
A  Technical  Reviews 
A  Program  Reviews 


ASR  -  Alternative  System  Review  ISR  -  In-Service  Review  PRR  -  Production  Readiness  Review 

CDR  -  Critical  Design  Review  ITR  -  Initial  Technical  Review  SFR  -  System  Functional  Review 

FCA  -  Functional  Configuration  Audit  OTRR  -  Operational  Test  Readiness  Review  SRR  -  System  Requirements  Review 

FRR  -  Flight  Readiness  Review  PCA  -  Physical  Configuration  Audit  SVR  -  System  Verification  Review 

IBR  -  Integrated  Baseline  Review  PDR  -  Preliminary  Design  Review  TRR  -  Test  Readiness  Review 
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Education  &  Training  (SYS  302) 


>SYS  302:  Technical  Leadership  in  Systems  Engineering 


•  Advanced  2-week  long  classroom  course  designed  for 
senior  DoD  acquisition  personnel. 


*  Focuses  on  the  application  of  technical  leadership  skills 
within  a  typical  DoD  SE  IPT  environment. 


*  Class  exercises  are  supplemented  by 
lessons  on  current  policy,  architectures, 
design  considerations,  ethics,  etc. 


*  Students  take  turns  leading  and  participating  in  an 
engineering  team  that  analyzes  and  resolves  a  variety  of 
technical  engineering  critical  issues. 


Introduction  to  Systems  Engineering 


Description  of  Systems  Engineering 


of  the  DoD  Acquisition  Framework 
Sfitially  reflect  an  underlying  engineering 
''process.  Bad  engineering  =  bad  products!  Unless 
properly  engineered,  no  affordable,  effective 
product  will  ever  reach  the  field. 

A  wide  variety  of  engineering  disciplines  have  to  be 
coordinated  and  properly  utilized  so  that  DoD 
systems  are  timely,  effective  and  affordable. 

System  Engineering  helps  ensure  that  that  happens. 

This  topic  defines  Systems  Engineering,  outlines  its 
scope  and  gives  examples  of  why  Systems 
Engineering  is  challenging. 

Select  NEXT  to  continue. 


r.j  .  f  ,  -  |  l  0l 


SYS-203 
Intermediate 
Systems  Planning, 
Research,  Development 
and  Engineering 

Part  2 


Requirements  and  System  Performance  i 

Input  Aggregation  determines  the  scope  of  the  effort; 
it  aggregates  all  available  inputs  based  on  the  exit 
criteria  and  outputs  of  Technology  Development.  These 
include  the  ICD,  CDD,  Acquisition  Program  Baseline 
(APB),  the  TEMP  and  SEP,  SDD  phase  exit  criteria, 
validated  maintenance,  and  support  concepts  and 
technologies.  These  will  govern  activities  in  this  phase. 

KPPs  have  been  definitized  in  the  CDD  and  used  as  part 
of  the  APB  to  formally  establish  the  'trade  space' 
available  for  design  decisions  during  this  phase.  The 
system  boundaries  are  clearly  identified,  as  are  key 
interfaces  and  information  exchange  requirements  with 
systems  or  host  platforms  external  to  the  system's 
boundaries. 


Using  primarily  Requirements  Development,  Input 
Aggregation  ensures  that  all  drivers  impacting  the 
system  design  are  completely  captured  in  the  System 
Performance  Specification,  which  forms  the  basis  for 
trade-offs  among  competing  parameters  (e.g.,  cost, 
schedule,  and  technical  performance)  that  can  be 
assessed  and  prioritized  against  program  goals  and  risk. 


Interpret  User 

Refine  System^ 
Performance  Spec  & 
Environmental  Constraints 


SYS  202 


Technical  Leadership 
in  Systems  Engineering 

April  2007 


*as  Of 

Sep  10,  2007 
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Learn.  Perform.  Succeed 


Education  &  Training  (LOG  204) 


>LOG  204:  Configuration  Management 

•  Fast-paced,  cross-disciplinary  course  that  provides  the 
knowledge  necessary  to  apply  configuration 
management  (CM) 

•  Includes  the  interrelationship  of  CM  to  such  life  cycle 
activities  as  systems  engineering,  data  management, 
logistics  support  planning,  and  weapon  system 
sustainment. 

•  Provides  an  overview  of  the  concepts  and  basic 
practices  of  CM,  including  configuration  identification, 
status  accounting,  audits  and  verification,  configuration 
change  management,  performance  measures,  and  CM 
planning. 
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Education  &  Training  (CLMs) 


>CLE  003:  Technical  Reviews 

•  Presents  essential  practical  guidelines  for  integrating  several 
different  technical  reviews  into  the  systems  engineering  process 
and  DoD  acquisition  life  cycle  based  on  best  engineering  practices. 

>CLL  008:  Designing  for  Supportability  in  DoD  Systems 

•  Provides  a  comprehensive  overview  and  introduction  to 
incorporating  the  principles  of  systems  engineering  throughout  the 
system  life  cycle  to  design,  develop,  produce,  and  sustain 
operationally  reliable,  supportable,  and  effective  systems. 

>CLE  017:  Technical  Planning  (Proposed  standard  for  FY  09) 

•  Presents  essential  and  practical  technical  planning  guidance  on 
how  to  integrate  program  management  tools,  such  as  earned  value 
management  and  risk  management,  with  systems  engineering  tools 
like  requirements  management,  technical  baseline  management, 
and  event-based  technical  reviews  into  an  effective  approach  for 
managing  programs. 
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SPRDE  Career  Field 

Update 


SPRDE  Career  Field  Update  Background 


>  Based  on  the  new  SYS  curriculum  and  the  FIPT  Chair’s 
proposal  to  enhance  certification  requirements,  the  SPRDE/SE 
FIPT  began  revising  certification  standards  and  Position 
Category  Descriptions. 

>  A  proposal  was  vetted  to  create  an  additional  career  path  to 
provide  maximum  flexibility  in  implementing  the  new  standards: 

•  Original  career  path  would  retain  the  1,  2,  4  years  of  experience  and 
similar  certification  standards. 

•  Additional  career  path  would  encompass  2,  4,  and  8  years  of 
experience  and  enhanced  certification  standards. 

>  The  Acquisition  Workforce  Senior  Steering  Board  accepted  this 
proposal  in  August  2006  and  the  implementation  details  were 
worked  out  and  resulted  in  an  agreement  in  February  2007. 
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Implementation  Details 


>  New  career  path,  “SPRDE/PSE  (Program  Systems  Engineer)”, 
with  a  new  position  code  and  position  category  description  was 
established  on  October  1, 2007. 


>  Targets  PEO  Chief/Lead  Engineer  and  Program  Systems 
Engineer  positions.  Requires  more  years  of  experience  and 
more  training. 


> 


>  Components  are  in  the  process  of  recoding  positions. 


No  change  to  existing  career  path, 
“SPRDE/SE  (Systems  Engineering)”. 

>  No  impact  on  current  SPRDE/SE 
certification  standards. 


18 


New  SPRDE/SE  &  IPSE  Career  Paths 
Certification  Standards 


Side-by-Side 

DoD  SPRDE/PSE  vs.  INCOSE  CSEP 


DoD  SPRDE/PSE 

INCOSE  CSEP 

Levels 

Three  levels 

Only  one  level 

Education 

Bachelors  or  graduate  degree  in  a  technical  or 
scientific  field  such  as  engineering,  physics, 
chemistry,  biology,  mathematics,  operations 
research,  engineering  management,  or  computer 
science. 

Bachelor’s  degree/equivalent  in  technical  field 
(Additional  experience  must  be  substituted  for  non¬ 
technical  degree) 

5  more  years  of  engineering  for  non-technical 

Bachelor’s  (total  10  years) 

10  more  years  of  engineering  if  no  Bachelor’s  degree 
(total  1 5  years) 

Experience 

Level  1:  2  years  of  technical  experience  from 
specified  career  fields 

Level  II:  4  years  of  technical  experience  from 
specified  career  fields  (in  acquisition  position) 

Level  III:  8  years  of  technical  experience  from 
specified  career  fields  (in  acquisition  position) 

5  years  minimum  in  multiple  SE  disciplines 

Training 

Several  acquisition,  systems  engineering,  and 
elective  courses  from  the  Defense  Acquisition 
University  (DAU),  based  on  level 

Only  what  is  needed  to  pass  the  exam 

Recommendations 

None 

At  least  3  Colleagues/Peers/Managers  who  are 
knowledgeable  in  Systems  Engineering 

Examination 

None 

(Exams  and  assessments  contained  in  individual 

DAU  courses.) 

Certification  exam,  based  on  current  INCOSE  SE 
Handbook.  Each  exam  costs  $80. 

Renewal 

None 

3  year  renewal 

120  Professional  Development  Units  required  during 
prior  3  years 

Renewal  Application  and  Fee  is  $100  -  discounted  to 
$50  for  INCOSE  member 

Continuing  education  log  required 

Certification  Cost 

None 

Application  fee  is  $400  -  discounted  to  $300  for 

INCOSE  members  20 

Core  Plus 
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What  is  Core  Plus? 


Core  Plus  is  an  enhancement  to 
the  AT&L  certification  framework. 

Core  Plus  is  designed  to  help  guide 
workforce  members  to 
additional  training  beyond  that 
required  for  certification. 

Core  Plus  Video: 

http://view.dau.mil/dauvideo/view/eventListing.jhtml?eventid=1583 
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Core  Plus  Target 


Common  acquisition 
foundation 
knowledge  and  skills 


Career  Field 
foundation 
knowledge  and  skills 


Plus”  or  job  competency 
point-of-need 
training  (often  CLMs) 
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Core  Plus  Development  Guide 

Example  for  SPRDE-SE  Level  II 


Core  Plus  Development  Guide3 

Type  of  Assignment 

Training  2 

Fund  Spec 

Software  /  IT 
Eng 

Dev  Eng 

S&T  Eng  /  Sci 

IRM  201:  Intermediate  Information  Systems  Acquisition  CiR 

X 

LOG  200:  Intermediate  Acquisition  Logistics ,  Pari  A 

X 

X 

LO  G  203 :  Reliability  and  Maintainability 

X 

X 

POM  201  A:  intermediate  PGM  Part  A 

X 

SAM  20 1 :  Intermediate  Software  A  equation  Management  C  R 

X 

STM  201 :  intermediate  S&T  Management  CR 

X 

TST  202:  intermediate  Jest  and  Evaluation  GR 

X 

X 

X 

X 

CLB  013:  intro  to  Earned  Value  Management 

X 

X 

C LB  01 7:  Performance  Measurement  Baseline 

X 

X 

CLC  041 :  Predictive  Analysis  and  Systems  Engineering 

X 

X 

CLE  007:  Lean  6  Sigma 

X 

X 

X 

CLE  018:  Outcome-based  Performance  Measures 

X 

X 

CLE  01 7:  Technical  Planning 

X 

X 

X 

X 

CLE  020:  Enterprise  Architecture 

X 

X 

X 

X 

C  LM  101:  Analysis  of  Alternatives 

X 

X 

X 

CLM  029:  Net-Ready  Key  Performance  Parameter  (NR-KPP) 

X 

X 

X 

X 

CLM  031:  improved  Statement  of  Work 

X 

X 

X 

X 

CLM  032:  Evolutionary  Acquisition 

X 

X 

X 

Education 


Graduate  degree  in  a  discipline  such  as  engineering,  physics,  chemistry,  biology,  mathematics,  operations  research,  engineering  management,  or  compjter  science. 


Experience 

2  additional  years  of  technical  experience 
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Core  Plus  Benefits  and  Challenges 


Benefits: 

•  Core  Plus  advances  the  AT&L  Competency 
Management  Model: 

-  The  right  learning  -  better  focus 

-  The  right  people  -  focused  on  competency  needs 

-  The  right  time  -  better  connection  to  job  needs 

-  Keeps  the  3-level  certification  framework 

Challenges: 

•  To  make  it  work,  Core  Plus  requires: 

-  Increased  Supervisor-Employee  interaction 

-  More  emphasis  on  Individual  Development  Plans 

-  Senior  leadership  support 
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The  Way  Ahead  for  SE  E&T... 


>  Keep  curriculum  up  to  date  and  properly  aligned 
with  revised  policies  and  guidance. 

>  Establish  two-way  communications  with  the  SE 
workforce  through  outreach  and  feedback. 

>  Enhance  SE  Communities  of  Practice  /  Websites. 

>  Work  with  academic  institutions  and  universities  on 
equivalency  issues  (i.e.,  AFIT  &  NPS). 

>  View  education  and  training  as  both  catalyst  and 
facilitator  for  cultural  change. 
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Questions? 
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for  Discussion 


>  How  do  you  facilitate  Cultural  Change? 

>  How  do  you  get  past  the  “Certification  Checklist” 
mentality? 

>  How  do  you  assess  the  SE  workforce? 

>  How  do  you  determine  if  education  and  training 
efforts  are  achieving  desired  outcomes? 


>  How  do  you  keep  the  SE  workforce  current? 

>  What  would  you  put  into  a  400-level  SYS  course? 
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Backup 


29 


OUSD  (AT&L)  Organization 
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Systems  and  Software  Engineering 


An  Organizational  Construct 
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What  We  Have  Done  To  Revitalize 
Systems  Engineering 


>  Issued  DoD-wide  SE  policy  -  focused  effort  on  up  front,  sound 
technical  planning;  issued  safety  policy 

>  Issued  guidance  on  Systems  Engineering,  test  and  evaluation  (T&E), 
software,  and  safety 

>  Worked  with  Defense  Acquisition  University  to  revise  Systems  Engineering 
curricula  --  currently  revising  T&E  and  enabling  career  fields  curricula 

>  Established  Systems  Engineering  Forum — senior-level  focus  within  DoD 

>  Integrated  development  testing,  software/system  assurance,  system  of 
systems,  and  open  systems  into  revitalization  efforts 

>  Instituting  a  renewed  emphasis  on  modeling  and  simulation 

>  Leveraging  closer  working  relationships  with  industry  and  academia 

>  Instituting  system-level  assessments  in  support  of  OSD  major  acquisition 
program  oversight  role 


Much  Accomplished  -  Much  to  Do! 
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Systems  Engineering  Plan  Trends 


>  What’s  working: 

•  Programs  beginning  to  establish  SE  WIPTs  early  in  the  life  cycle 
to  develop  and  document  their  technical  planning 

•  Increased  Program  Executive  Office  level  Lead/Chief  Systems 
Engineers  involvement  in  SEP  development 

•  Movement  to  event-driven  versus  schedule-driven  programs 

-  More  focus  on  entry  and  exit  criteria  for  technical  reviews 

>  What  needs  work: 

•  Firming  up  technical  planning  prior  to  RFP  release 

•  Proposed  processes  for  a  program  not  always  tailored  to  fit 
program 

-  Often  appear  to  be  copied  from  a  manual  or  guide. 

•  SEP  author  is  someone  in  program  office  (contractor  or  junior 
person)  who  is  not  familiar  with  the  technical  strategy. 

•  SEPs  need  to  be  consistent  with  key  program  documents 
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What’s  Next? 


>  We  have  revitalized  Systems  Engineering  Policy,  Guidance, 
Education  and  Training... 

>  We  have  driven  good  systems  engineering  practices  back  into  the 
way  the  acquisition  community  does  business,  and  have  had  a 
positive  impact  on  programs... 

>  We  have  expanded  the  boundaries  to  include  increasingly  important 
enablers  for  sound  SE  application... 

>  We  have  a  rigorous  process  to  capture  what  went  wrong... 

>  ...but  failed  to  change,  root  cause  behavior  that  leads  to  programs 
that  do  not  meet  cost,  schedule,  and  performance 
expectations... adequate  maturity  at  program  initiation 


Much  Accomplished  -  Much  to  Do! 
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vector/CSP 


Transforming  USCG  Logistics 


A  systems  engineering  approach  to  transforming  the  USCG  Enterprise 
Logistics  Systems 


Patrick  W.  Cumby 

Director  Performance  Systems 

VectorCSP 


www.  vectorcsp.  com 


vector/CSP 


Agenda 


•  USCG  Logistics  Transformation 
Background 

•  Enterprise  Transformation  Basics 

•  USCG  Logistics  Transformation 

•  Demos  of  Logistics  Models  and 
Transition  Dashboards 
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vector/CSP 


Logistics  Transformation  Background 


•  USCG  has  several  "stovepiped"  logistics 
business  models  (surface,  air,  shore,  IT) 

•  Models  have  evolved  over  time  and  are  not 
integrated  or  strategically  aligned 

•  Some  of  the  various  models  utilize  modern 
logistics  concepts,  others  do  not 

•  Limited  visibility  into  systems  performance 

•  Limited  ability  to  manage  costs  and 
effectiveness 

•  Then  along  comes  the  Deepwater 
program,  the  CFO  Act,  and  Katrina... 
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vector/CSP 


Logistics  Transformation  Drivers 


•  Deepwater  program  -  recapitalization 
of  USCG  assets  and  capabilities 

-  Deepwater  program  has  experienced 
several  issues  that  have  led  to  a  major 
restructuring  of  the  program. 

•  CFO  Act  -  Mandate  to  institute  total 
asset  visibility  and  financial  controls 

•  Success  of  Katrina  disaster  response  - 
demonstrated  strengths  of  USCG 
Aviation  logistics  model 
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Logistics  Transformation  Objectives/  Scope 


vector/CSP 


•  Admiral  Allen  issues  Cl  AO  #4 

•  Bi-level  maintenance  w/more  standardized 
procedures. 

•  Centralized  supply  chain  management  w/spending 
driven  by  maintenance  requirements. 

•  Disciplined/standard  Coast  Guard-wide  engineering 
and  logistics  business  processes,  modeled  after  our 
internal  best  practices  currently  in  use  in  aviation. 

•  Strong  configuration  management  processes, 
w/associated  compliance  inspections,  to  ensure  all 
configurations  are  safe,  effective,  and  supportable 
when  installed. 

•  Reduce  the  number  of  financial  and  information 
systems. 
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Transformation  Support  Team 


•  Logistics  T ransformation  Program 
Integration  Office  (LTPIO)  established 

•  General  Dynamics  contracted  to 
provide  program  management 

•  VectorCSP  contracted  to  support 
organizational  and  logistics 
transformation 


Transformation  Management  Approach 


•  LTPIO  chose  VectorCSP's  Pathfinder 
approach  to  develop  the  logistics 
business  model 

•  Pathfinder  is  a  systems-oriented, 
tools- based  transformation 
management  methodology 

•  Pathfinder  incorporates  a  complete 
business  systems  performance  model, 
with  an  emphasis  on  behavioral 
engineering 
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Let's  go  back  to  Org 
Transformation  Basics! 


Start 

Here 


^  Finish! 


8 


vector/CSP 


Enterprise  Transformation  ABCs 


A:  Where 

B:  Where 

you  are  now. 

you  want  to 
be. 

A 


C:  The  path 
from  A  to  B. 


B 


A:  Where  you  are  now. 


•  I  n  order  to  get  from  A  to  B,  you 
have  to  understand  A. 

•  A  is  your  "As-ls"  organizational  and 
business  model 

•  A  is  usually  very,  very  complex... 


Usually  the  important 
cultural  and  political 
aspects  of  a  business 
model  are  not 
well  documented. 
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B:  Where  you  want  to  be. 


I  n  order  to  get  from  A  to  B,  you 
also  have  to  understand  B. 

B  is  your  'To-Be"  organizational  and 
business  model 

B  is  usually  not  well  defined... 
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C:  The  Path  from  A  to  B 


The  really, 
really,  really 
hard  part. 


— i —  c 


V: 

HL 


i  ~i .  i 
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Question: 

What  are  the 
roadblocks 
along  this  path? 
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Speedbumps  on  the  Path 


Strategic  Alignment 

Technology 

Process 


Question: 

Of  these 

roadblocks,  which 
is  most  difficult 
to  overcome? 


Organizational  Design 

Culture 

Politics 


ur^H- 


i  i — i — i 


tI — 


Path  from  A  to  B 
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To  sum  it  up. 


You  move  from  a  highly 
complex  (and  usually 
broken)  system... 


...through  a  complex 
transformation  process 
fraught  with  cultural, 
technical,  and  political 
barriers... 


T@§®lt 

®lil 


...to  get  to  what  is 
usually  a  poorly 
understood  end  state. 


« 


Question: 

What  percentage 
of  enterprise 
transformations 
succeed? 
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I  t's  no  wonder  that 

85% 

of  all  enterprise  transformations 

Do  not  fully  succeed! 


vector/CSP 


What  you  need  to  succeed 


1.  Clearly  defined  transformation  objectives 

2.  A  way  to  identify  A  and  B 

3.  A  roadmap  to  transform  the  structures, 
processes,  technology,  culture  and 
politics  of  the  organization 

4.  A  way  to  manage  the  astounding 
complexity  and  mountains  of  data  of 

such  a  large-scale  endeavor 

5.  A  way  to  communicate  with  all 
stakeholders 

6.  A  way  to  measure  success  of  the 

transformation 

7.  A  fully-dedicated  transformation 
management  team 
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I  n  other  words... 


...you  need  a  systems  engineering 


The  Coast  Guard  Logistics 
T ransformation 


Start 

Here 


^  Finish! 
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Coast  Guard  Transformation 


A:  Separate 
logistics  models 
for  surface  and 
aviation. 


A 


C:  Logistics 
T  ransformation, 


Path  from  A  to  B 


B:  Standard 
logistics 
model  based 
on  aviation 
model. 


B 


Desired  State 


Current  State 


Multiple  logistics  models  (naval,  aviation, 
shore,  C4ISR) 

Problematic  logistics  systems  for  naval, 
shore,  and  technology  systems 

-  Non-standard  fleet  assets  and  inventories 

-  Antiquated  logistics  processes  and  technology 

-  Sub-optimal  acquisition  model 

-  Poor  financial  controls 

•  Not  compliant  with  CFO  Act 

•  Problems  with  Deepwater  program 

•  Getting  the  mission  accomplished  despite 
sub-optimal  logistics  systems  due  to 
dedication  and  "can-do"  attitude 
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B:  The  'To-Be"  State  of  CG  Logistics 


Adopt  CG  Aviation  I  nteg rated 
Logistics  Systems  (ILS)  model 

Standard  fleet  assets  and  Total  Asset 
Visibility 

Integrated  technology  infrastructure 
(based  on  modified  ALMIS) 

Transparent  and  tightly  controlled 
financials 

CFO  Act  Compliance 

Systems  measures  of  effectiveness 
( MOEs) 


C:  The  Path  to  Transition 


Commandant's  Cl  AO  establishes 
transformation  objectives 

LTPIO  established  as  the  fully- 
dedicated  transition  team 

Pathfinder  Performance 


Modeling  approach  chosen  by 
LTPI O  as  a  key  transformation 
management  tool. 


About  Pathfinder 


vectoiytSP 

•  Pathfinder  is  a  dedicated 

transformation  modeling  and 

support  system 

•  Designed  for  large-scale 

organizational  transformations 

•  Pathfinder  is  based  upon  a  systems 
engineering  approach  to  org 
transformation 
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Performance  System  Engineering 


vectoiyCSP 

•  Pathfinder  breaks  organizational 
performance  into  discrete  systems  elements 

^  V  "C  ^  %■  . . . 

•  It  incorporates  process  engineering, 
organizational  design,  enterprise 
architecture,  and  most  importantly 
behavioral  engineering 

HH  •  It  enables  modeling  of  all  elements  that 

influence  organizational  performance 

•  It  makes  it  possible  to  manage  the 
complexity  of  a  large  transformation 


24 


Performance  Systems  Components 


Key  building  blocks  of  a 
Performance  Model 

System 
System  Outcome 
^  J  ob  Role  or  Team  (People) 
Strategic  Objective 
4^  Policy 

Technology 
Organization  ^ 
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Question: 

Which  of  these  elements 
is  most  critical  to  performance, 
yet  most  often  overlooked? 
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Performance  Model  Basics 


A  performance  system  can 
have  multiple  subsystems 


A  performance  system  is 
defined  by  its 
System  Outcomes 


/  - 
System 
Outcome 


The  Outcome  is  the 
central  element  of  a 
performance  model  26 


Examples  of  Systems  and  Outcomes 


•  The  Coast  Guard  logistics  performance  model  is 
based  on  the  standard  I LS  ( I  ntegrated  Logistics 
System) 

•  Excerpt  from  the  CG  model: 


-J  ILS 

- ±J  Maintenance 

I - Supply 


*4 


Cost  driver  components  identified 
Part  enrolled  in  ALMI S 
Level  of  Repair  determination 
Many  others... 
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M 


Coast  Guard  I LS  System  Hierarchy 


Design  I  nterface 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 


Maintenance 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 


Manpower 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 


Supply 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 


Support  Equip. 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/ OGA  Contracts 


Tech  Data 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/OGA  Contracts 


Training 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/OGA  Contracts 


IT 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/OGA  Contracts 


PHS&T 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/OGA  Contracts 


Facilities 

Ops 

Manpower 

Facilities 

Training 

Tools/ Equipment 
I  nfo  Management 
Tech  data 

Envi  ronment/ Hazmat 

CM/Standardization 

Safety 

Comms/  Feedback 
Finance 

Vendor/OGA  Contracts 


Performance  Model  Relationships 


All  components 


What  job  roles  are  involved  in 
producing  the  outcome? 


What  policies  are  related 
to  the  outcome? 

What  strategic  objectives  does  the 
outcome  support? 


What  organizations  are  involved 
in  producing  the  outcome? 


What  technology  is  involved  in 
producing  the  outcome? 


What  systems  are  related  to 
the  outcome? 
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Modeling  A  and  B  for  CG  Logistics 


•  This  "performance  modeling" 
approach  is  used  to  create  as- is  (A) 
and  to-be  (B)  models  of  CG  logistics 


•  The  next  step  is  to  develop  the 
Logistics  Transformation  Mode!  (C) 
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Outcomes  as  Key  Transformation  Drivers 


"To-Be" 

System 

Outcome 


I  n  the  Pathfinder  transformation 
model,  system  outcomes  are  the  key 

transformation  drivers. 

in  other  words,  if  the  “to-be" 
outcome  can  be  achieved,  then  the 


transformation  to  that  part  of  the 
system  is  considered  a  success 
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Eliminating  Outcome  Performance  Gaps 


There  are  typically  gaps  between  an  as- is 
outcome  and  the  to-  be  outcome 

These  gaps  are  recognized  in  our  model  as 

a  performance  gap  element 

Some  gaps  are  simply  differences  in 
processes  or  personnel 

Other  gaps  may  require  technology, 
infrastructure,  or  organizational  changes  to 
eliminate 

These  gaps  must  be  eliminated  by  actions 

called  performance  interventions 


Elements  of  the  Transformation  Framework 


•  Pathfinder  includes  gaps  and  interventions  as 
discrete  elements  of  the  transformation  model 


As-ls 

System 

Outcome 


To-Be 

System 

Outcome 


Performance 
I  intervention 


Performance 
I  ntervention 


Coast  Guard  I  intervention  Examples 


•  Outcome:  Approved  Maintenance 
Procedure  Card  produced  I  AW 
COMDI NST  xxx.x 

-  Gap:  Maintenance  Requirements 
List  (MRL)  not  defined  for  surface 


•  I  intervention:  Perform  MSG-3  logic  analysis 

•  I  ntervention:  Enroll  asset  in  ACMS 

•  I  ntervention:  Modify  MPC  process  guide 

•  Intervention:  Identify  and  train  MPC 
production  staff 
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About  I  interventions 


vectoi^CSP 

•  Each  intervention: 

-  I  s  an  actionable  item 

Performance 

intervention  -  Has  an  accountable  owner 

-  Has  schedule  constraints 

-  Has  associated  resources 

-  Can  be  used  to  build  a  transition  project 
plan 

-  And  most  importantly,  has  measurable 
success  criteria 


Putting  the  elements  together... 


•  Strategies 

•  Systems 

•  Subsystems 

•  Outcomes 

•  I  nf I uencers  (policies,  organizations, 
people,  technology,  etc.) 

•  Gaps 

•  Interventions 

•  How  does  it  all  fit  together? 
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Transformation  Model  Framework 


*4 


■C'l 


Strategic  Framework 

Systems  Framework 

Org  Framework 

Technology  Framework 

Org  Unit 


Perfornner: 


IT  System 


IT  Feature 


J  ob  Role 
Team 
Automated  System 


Faci  I  ity/  Equi  pment 


Facility 
Tool/ Equi  pment 


Transition  Framework 


Transformation  Goal 


Performance  Gap 


Transformation  Objective 


I  ntervention 


Transition 
Success  Criteria 


The  USCG  Transition  Process 


vectoiytSP 

•  LTPIO  alignment  conducted,  transformation  objectives  identified 

•  Docs  and  resources  reviewed 

•  SMEs  identified 


I LS  systems  outcome- based  framework  developed 

Best  practices  identified  by  SMEs  (42) 

Preliminary  outcomes  identified  by  SMEs  (800) 

Org  and  strategic  models  defined 

Key  outcomes  identified  (250) 

Framework  relationships  to  key  outcomes  identified  by  SMEs  (docs,  job 
roles,  policies,  best  practices,  etc.) 

Skilled  Performers  (SPs)  identified  by  USCG 

SP  outcome  review  worksheets  prepared 

SP  outcomes  reviews  and  validation  conducted 

All  data  collected  in  Pathfinder  Performance  Modeler 

All  data  reviewed 

Analysis  reports  prepared 

PVs  and  Pis  developed  using  facilitated  meetings  with  LTPIO  working  groups 
Activity  crosswalks  prepared 

Project  plans  and  transition  dashboards  developed 
Transition  training  prepared  and  delivered 
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800  aviation  logistics  model  outcomes  were 
identified  in  all  10  I LS  elements  (and  a 
cross-cutting  set  of  subelements) 

250  Key  Outcomes  (transformation  drivers) 
identified 

Key  Outcomes  mapped  to 

-  41  Systems 

-  698  J  ob  roles,  teams,  or  org  units 

-  80  Documents  (Policy/Directives/Statutes,  etc) 

-  122  IT  systems  and  features 

-  84  Strategic  elements  (goals  and  objectives) 

-  Over  600  functional  and  business  requirements 

-  Over  7,000  performance  factors  and  influencers 

Nearly  22,000  performance  relationships 
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Model  Configuration  Management 


Aviation 
Logistics  WPM 
Configuration 
Management 


o 

Update 

The  WPM  database  is 

updated. 

Change 


The  WPM  administrator 
makes  the  CCB-approved 
change  to  the  WPM. 


© 


CCB 


WPM  change  reviewed  and 
approved  via  CCB  and 
CG-22. 


This  is  a  high-level  overview  of  the 
proposed  Aviation  Logistics  WPM 
configuration  management  process 
which  can  be  used  both  to  validate 
and  enhance  WPM  content  and  also 
to  manage  its  configuration  overtime. 


People  are  Key 


•  Changing  the  behavior  of 
your  people  is  the  most 
difficult  task  of  all... 


•  Culture  and  politics  are  the  most  difficult 
roadblocks  to  logistics  transformation 


You've  got  to  convince  people  at  all  levels 
to  change  the  way  they  operate. 
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Promoting  cultural  and  political  change 


•  Each  outcome  identifies  human  factors 
influencers  (technological,  environmental, 
social,  and  process) 

•  Relationships  in  model  indicate  cultural  and 
political  power  bases 

•  Model  enables  stakeholders  to  "see"  vision 

•  The  Transformation  Dashboard  is  key 

to  producing  measurable  changes  in 
organization,  infrastructure,  processes, 
systems,  and  behavior. 

•  Manages  and  measures  progress  in 

making  the  changes  required  to 
achieve  “to-be"  outcomes 
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Transition  Success  Criteria 


To-Be 

System 

Outcome 


Each  level  includes  a 
summary  scorecard  that 
shows  transition  progress. 


Performance 

Gap 


Performance 
I  intervention 


External  audit  conducted  -  Final  Operating  Capability  (FOC)  achieved 
Internal  audit  conducted  -  Initial  Operating  Capability  (IOC)  achieved 
^  Process  requirements  identified  and  plan  of  action  in  place 
No  Progress 
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•  Outcome:  Authorized  Chemical  List 

-  I  ntervention:  Chemical  Locker  Storage 
Established 

-  Criteria: 

•  Red:  No  Progress 

•  Orange:  Location  for  chemical  storage  locker 
identified. 

•  Yellow:  Authorized  Chemical  List  established  and 
utilization  instruction  developed  and  signed  by  Sector 
Engineering  Officer. 

•  Green:  Personnel  trained  in  proper  storage 
procedures  and  usage  of  the  Chemical  locker  I  AW 
Hazmat  plan. 

•  Owner,  schedule  criteria,  compliance  inspection 
process,  etc :  defined  for  intervention  in  mode! 


Sample  USCG  Scorecard 
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Summary 


•  Performance  Model  Framework  identifies  all 
elements  of  logistics  system  performance 

•  All  relationships  between  systems  elements 
are  defined 

•  Performance  outcomes  are  central  to  model 

•  Each  outcome  has  transition  plan  that 
identifies  gaps  and  interventions 

•  Each  intervention  has  measurable  success 
criteria 

•  Success  criteria  form  the  basis  for  the 
Transition  Dashboard 

•  T ransition  Dashboard  drives  systems, 
organizational,  technological,  and 
behavioral  change 
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Demos  and  Questions 


•  USCG  Logistics  Performance 
Management  System  (LPMS) 

•  Logistics  T ransition  Dashboard 

•  Transformation  support  materials 


r  Experiences  in  Applying  SysML  to 
Develop  Interoperable  Torpedo  Modeling 
L  and  Simulation  Components 


Presented  to: 

NDIA 

10th  Annual  Systems  Engineering 

Conference 

San  Diego,  CA 

Presented  by: 

Thomas  Haley 

Naval  Undersea  Warfare  Center 
Division  Newport 
Newport,  Rl 


OPEN  S  YSTEMS 


JOINT  TASK  FORCE 


yroioiioiooii 
(010100101 
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ADVANCED 
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24  October  2007 


Outline 


•  SysML  Case  Study  Motivation 

•  TEAMS  Project  Background 

•  SysML  Proof  of  Concept 

•  Lessons  Learned 

•  TEAMS  Perspective:  SysML  Pros  and  Cons 

•  Acknowledgements 


fEAM$ 


Motivation :  fespBp?!% 

Feasibility  of  Open  Standards 

•  Funded  by  Office  of  Secretary  of  Defense, 
Systems  and  Software  Engineering 

•  Determine  if  open  standards  can  be  used  to 
describe: 

-  System  of  systems  (SoS)  architectures  based  on 
computer  models 

-  System  components  as  elements  of  composable 
distributed  simulations 

•  Determine  whether  SysML  models  can  be 
used  in  conjunction  with  performance 
simulation  models 


^RPEDOENTEfiPfi^ 


ADVANCED 


JOaiNG&SIMULW^- 


Background: 
TEAMS  Simulation  Scope 


Military  M&S  Resolution  Levels 


TEAMS  Emphasis: 
“Launch-to-Hit” 
Analysis 


TEAMS:  Torpedo  Enterprise  Advanced  Modeling  &  Simulation 


ncevCE  OF  NAWL  . 


Background: 

High-Level  M&S  Requirements  » 


Torpedo  M&S  Components 
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TEAMS  Background  fUgt 


•  Problem:  Modeling  &  Simulation 
Business  “Model”  Obsolete 

-  Monolithic 

-  Stove  pipes 

-  Single  developers 

-  No  communication 

•  Solution:  Foster  Collaborative 
M&S  Development  Environment 

-  Standardize  M&S  architecture 
framework  and  component 
models 

-  Reduce  the  technology 
development  timeline 

-  Increase  model  content, 
implementation  efficiency  and 
reuse 

-  Reduce  cost 
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Overall  TEAMS  Goals 


•  Modeling  and  Simulation  Community  Collaboration 

•  Standardized  architecture  framework 

-  Conceptual  reference  model 

-  Model-based  requirements  specifications 

•  Standardized  reference  model  interfaces 

-  Interchangeable  &  composible  components 

-  Extendable  to  other  applications  (e.g.,  XML  schema) 

-  Semantically  described  (e.g.,  OWL  ontology) 

•  Document  standards  and  requirements 

•  Cost  effective  process  to  achieve  interoperability  and 
composability 

•  Business  model  for  future  cross-organization  M&S  funded 
efforts 


OfPjCEOFNAMJLRES^Qy 
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TEAMS  Core  Requirements 


Q^pC£OFNAMALRE$£^Q^ 

r  ^rTTmonPB^* 
^1011010011^^3 
!1  JlJ^iCADVANCEDjSr 


iJS°&ING&  SIMULA^ 


1.  Standard  Interfaces 

2.  Platform  Independence 

3.  Open  Standards 

4.  Model  Realizable  Systems 

5.  Extensible  Interfaces 

6.  Evolving  Standards 

7.  Loosely  Coupled  Interfaces 

8.  Tiers  of  Interfaces 

9.  Support  Different  Levels  of  Detail 

10.  Standard  Implementation  Strategies 


8 


ncPVCE  OF  NAVAL  RESSto^, . 
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Organizations  Looking  to  TEAMS  TUm’ 


THE  Op  en  group 


International  organization,  developers  of  TOGAF  architectural 
framework 

-  Wants  TEAMS  as  test  case  for  TOGAF  8.1.1  and  9.0 

-  Interest  in  using  TEAMS  to  test  synergy  between  DoDAF  and  TOGAF 
frameworks 

-  Wants  TEAMS  for  its  process  to  incorporate  Ontologies 
(relationships  of  components) 


OBJECT  MANAGEMENT  GROUP 


International  organization,  developers  of  several  business 
communications  standards 

-  Used  TEAMS  as  test  case  for  their  TOGAF/  Model  Driven 
Architecture  (MDA)  under  the  TOGAF/MDA  Synergy  Project 


OPEN  SYSTEMS 


JOINT  TASK  FORCE 


The  Open  Systems  Joint  Task  Force  of  the  Office  of  Secretary  of 
Defense  (OSD) 

-  Wants  to  convert  TEAMS  UML  artifacts  to  the  newly  approved 
SysML  standard  to  demonstrate  utility  of  the  new  standard 
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TEAMS  is  quickly  yielding  highly  visible  and  transitionable  results. 


High-Level  Process: 

TOGAF ADM 


QfpCEOFNAMJLRES^Qy 


JR$ 


The  Open  Group: 

IT  Consortium 
Offers  Consortia  Services 

TOGAF: 

The  Open  Group 
Architecture  Framework 

ADM: 

Architecture 
Development  Method 


“OMG™  is  [a]  ...  not-for-profit  computer  industry 
consortium  ...  developing  enterprise  integration 
standards  for  a  wide  range  of  technologies  [.../...] 
industries  ...  enabling]  powerful  visual  design, 
execution  and  maintenance  of  software  and  other 
processes...” 

•  CORBA  -  Common  Object  Request  Broker 

•  UML  -  Unified  Modeling  Language 

•  SysML  -  Systems  Engineering  Modeling  Language 

•  Numerous  others  in  diverse  industries  (e.g.,  business) 

•  Developer  of  Model  Driven  Architecture  (MDA)  method 
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OMG  has  a  model-based  emphasis  in  developing  standards 


UML 


QfpCEOFNAMJL^S^Qy 
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TIME  Consists  of  13  Diagrams 


Structure:  E.g.,  Class  Diagram 


Behavior:  E.g.,  Activity  Diagram 


Interaction:  E.g.,  Sequence  Diagram 

OMG  models  are  MOF-Based  -  Meta-Object  Facility  Standard 

Think  “TurboTax” 


1] 


F>*wCmi4  dowrtaicf  "-j-1 


1] 


V 


Using  MDA  in  SE  Context 


Q^pC£OFNAMALRE$£^Q^ 
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Baseline  Technology! _ 

Architecture  "llfgi 


Weapons  Analysis  Facility  (WAF) 


WAF  Architecture 

Origin-Based  Sim  ulators _ 


Technology  Requirements  Model  (TRM) 

TRM’s  CARLEE 


Conceptual 

Reference 

Model 


ENVIRONMENT AL  ACOUSTICS 

OBJECT  ACOUSTICS 

WEAPON/PLATFORM  ACOUSTICS 

Sound  Propagation 

Active  Target  Strength 

Self-Noise 

Multipaths 

Target  Radiated  Noise 

Radiated  Noise 

Ambient  Noise 

Countermeasures 

Beam  Forming 

Reverberation 

Magnetics 

Pulse  Types 

Surface  and  Bottom  Types 

“Artificial”  Targets 

Wake 

Platform  Sensors 

Ice 

Remote  Sensors 

OTHER 

Command  and  Control 

Propulsion 

Propulsors 


SST  Component  Models 


ORB  IS  Object  Examples 


Sonar  System  Toolset  (SST) 


The  Method: 


Model  Driven  Architecture  (MDA) 


Artifact  Type 


Skilled 
Transformation 


MDA 

Computational 
Independent 
Model  (CIM) 


Automated 

Transformation 


Automated 

Transformation 


TEAMS  Conceptual  Reference  Model 


MDA  Platform 
Independent 
Model  (PIM) 


MDA  Platform 
Specific  Model 
(PSM) 


Working 


Software 


8.  Environment 

•  Sound  Velocity  Profile  (SVP) 

•  Surface  Wave 

•  Bottom  Characteristics 

•  Boundary  Characteristics 

•  Bathymetry 

•  Bottom  Scatter  Strengths 

?  Environmental  False  Taraets 


9.  Model  Description 

•  Rdelity 

•  Level  of  Detail 

•  Validity 

•  Launchers 

•  Submarine  and  Surface  Ship 
Classes 

•  Inter- platform  Communication 
(relationships) 


1 .  Propagation 

•  Ray  Tracing 

•  Bottom  Scattering 


2.  Platform/Vehicle 
and  Tracking 

•  Location 

•  Orientation 

•  Tlme/Space/Fbsition 
Information  (TSPI) 

•  Kinematics 


3.  System 
Components 
(Platform/Torpedo) 

•  Propulsion 

•  Sonar 


5.  Targets 

•  Highlights 

•  Active  Sources 

•  Non-Acoustic 


4.  G&C- Signal 
Processing  Chain 

•  Command  and  Control 

•  Tactics 


6.  Data  Interchange 

•  R-ecision 

•  Units 

•  Brors 

•  Tolerances 

•  Uncertainty 


7.  Simulation  Run  Info 
&  Management 

•  Time 

•  R/ents 


TEAMS  Conceptual  Reference  Model 


8.  Environment 


Sound  Velocity  Profile  (SVP) 
Surface  Wave 
Bottom  Characteristics 
Boundary  Characteristics 
Bathymetry 

Bottom  Scatter  Strengths 


1 .  Propagation 


Ray  Tracing 
Bottom  Scattering 


9.  Model  Description 

•  Fidelity 

•  Level  of  Detail 

•  Validity 

•  Launchers 

•  Submarine  and  Surface  Ship  Classes 

•  Inter-platform  Communication 


7.  Simulation  Run 
Info  &  Management 


6.  Data  Interchange 


Time 


Events 


Precision 

Units 

Errors 

Tolerances 

Uncertainty 


Wditiwcl 


Conceptual  Level 


Platform 

Diagram 


ifljy 
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HMU 
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Environment 
Conceptual  Level  Diagram 
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The  Method: 
Model  Driven  Architecture  (MDA) 


Artifact  Type 


TEAMS  UML  Component  Diagrams 


Reference  Implementations 
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TRM  Propagation  Tool 
‘PETE’ 


Unddrteo  M'^dium 


Drietl 


;  :T ..  .1"!':  “ 

Output 


TEAMS  Demonstration 
Undersea  World 


1  rpik 


IjajriCiO 


PlatTomi  fT&rpwte) 


prauH 


Plafllgrm 


iWiKWlTi^J 


PEadorm  (Tang*tj 


lrHf-n«JT- 


TEAMS  PSM: 
Implementation 


T’eam* 


NAVOCEANO 
SIPRNET  Web  Site 


Closed-Loop  SimuLink™  Torpedo, 
Environment  &  Target 


Applied  Physics  Lab 
University  of  Washington 


PMS  404 


UNDERSEA  WEAPONS  PROGRAM  OFFICE 


Proof  of  Concept 

•  Port  existing  UML  to  SysML 

-  Torpedo  system  components 

-  Simulation  environment 

•  Extend  TEAMS  SysML  to  include: 

-  Requirements  traceability 

-  Parametrics  and  constraints 

•  Share  experiences  and  lessons  learned 
using  SysML  for  architecture  and 
component  modeling 


OfPjCEOFNAMJLRES^Qy 


team® 


TEAMS  SysML 
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to  SysML  Approach 

•  Convert  UML  Class  Diagrams  to  SysML 
Block  Definition  Diagrams  (BDDs) 

•  Convert  UML  Component  Diagrams  to 
SysML  Internal  Block  Diagrams  (IBDs) 

•  Represent  Behavior  Relationships  Between 
Blocks  as  Activity  Diagrams  (new!) 

•  Capture  Requirements  Traceability  (new!) 

•  Capture  Parametric  Relationships  and 
Constraints  (new!) 


fEAM* 
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TEAMS  Perspective: 
SysML  Pros  and  Cons 


Cr_^vy^-  Li  .  ;T  — - 


Pros 


Cons 


Requirements 

-  Explicitly  lay  out  requirements  and 
consequences 

Views  and  Viewpoints 

-  Can  separate  requirements  and  model 
views  based  on  stakeholders  concerns 

Structure 

-  Ability  for  model  structure  to  verify 
requirements 

•  Can  search  for  requirements  that  aren’t 
verified 

•  Can  search  for  model  components  that 
aren’t  justified 

-  Separation  of  structure  from  behavior 

•  SysML  BDDs  vs.  IBDs  and  Activities 
allow  for  clear  separation 

•  UML  allows  this,  but  easier  to  implement 
in  SysML 

Behavior 

-  Dashed  line  for  activity  flow  is  more 
aesthetically  pleasing 

•  vs.  UML  solid  line 


Allocating  CIM  to  PIM 

-  Difficulty  with  abstract  activities 

•  Exit  path  dependent  on  logic  within  an 
activity  is  not  accessible  and  can’t  be 
modeled 

•  Not  represented  well  in  either  UML  or 
SysML  -  tactical  controller  example 

Implementing  PIM 

-  Not  “direct”  for  some  SysML  features 

•  Flow  ports,  continuous  activities, 
parametric  constraints  involve  more 
components  than  just  themselves 

•  Flows  in  “real  systems”  easier  to 
represent 

•  Flows  in  software  modeling  are  open  to 
interpretation 

-  Requires  additional  documentation  of 
model  to  bridge  between  SysML 
feature  and  executable  code 


TEAMS  Perspective: 

SysML  Pros  fWKGfr 


Pros 


Requirements 

-  Explicitly  lay  out  requirements  and  consequences 

Views  and  Viewpoints 

-  Can  separate  requirements  and  model  views  based  on  stakeholders  concerns 

Structure 

-  Ability  for  model  structure  to  verify  requirements 

•  Can  search  for  requirements  that  aren’t  verified 

•  Can  search  for  model  components  that  aren’t  justified 

-  Separation  of  structure  from  behavior 

•  SysML  BDDs  vs.  IBDs  and  Activities  allow  for  clear  separation 

•  UML  allows  this,  but  easier  to  implement  in  SysML 

Behavior 

-  Dashed  line  for  activity  flow  is  more  aesthetically  pleasing 

•  vs.  UML  solid  line 


Sponsor  Requirements 


Reduced  Duplicate  Efforts 

notes 

LXfferent  contractors  should  not  have  to 
research  the  same  technology  or  enabling 
model  in  order  te  accomphsh  therr^ecrtfc 
goate.  to  stead ,  similar  efforts  should  j&e 
merged  together  and  the  result  shared . 

Less  Component  Integration  Time 

notes 

Component  developers  should  be  aAte  te 
spend  their  time  and  resources  on 
developing,  and  be  able  te  vehfyhew 
ideas  with  simulation  quickly. 

Contractor  Interoperability 

notes 

ff  two  different  contractors  write  two 
different  components,  they  should  be  able 
te  communrcate  with  each  other. 

Model  Realizable  Systems 

notes 

Component  developments  need  to  be 
convertible  into  a  real  system  to  be  useful. 

Reuse  Legacy  and  New  Components 

notes 

Some  mechanism  should  enable  older 
systems  to  be  pulled  into  simulations  with 
ne wv  interfaces,  and  newly  developed 
components  should  have  some  easily 
reusajbte  interface  to  reduce  this  problem  in 
the  future. 

Room  for  Future  Growth 

notes 

Adaptivity  to  future  changes  is  important  in 
any  large  initial  investment,  rnctedrng 
standardization  of  components.  There  is  a 
risk  of  the  standards  jberng  out  o^  date 
before  they  have  enough  time  to  be  useful. 

Rationale  for  Deriving 
TEAMS  Core  Values 
onsor  Requirement(s) 


^Q^EDOENTEfiP^il^ 
^rtrrrFfook1 1 1  ™ 
r_A_^J^1011010011  E 
>^/Gioiooioi2fT^VANrFD-i 

^GlSIMUlW^? 

teams 


C  o  ntra  cto  r  Interoperability 

ff  fw'o  different  contractors  write  two 
different  components,  they  should  /be  a/ble 
to  communicate  with  each  other 


Standard  Interfaces 


TEAMS  will  provide  standardized  interfaces 
for  simulation  elements  as  well  as  platform 
components  Any  developer  of  either  system 
claiming  TEAMS-Compliance  to  any  degree 
will  use  these  interfaces  as  the  external 
i boundary  of  their  implementation . 


a  rationale®  [\ 

By  standardizing  the  approaches  used 
by  all  contractors  when  developing  their 
implementations,  contractors  can 
depend  upon  the  defined  interfaces 
without  even  knowing  who  will  be 
writing  it. 


(from  Core  Acquirements) 


Open  Standards 


Standard  FSM  Iffipkmentation^Stiategies 

noi&£ 

To  promote  rntwchmge  of 
ccwpweate.  T£AitfS  wMpiouiti* 
CQfHfiBrimpteMmtM ok  This  will 

an  floss  of  CQtitfltumcati  on  a'ue  to 
drtfemM  Mappings  to.it  ffr*  TEAMS  platfbm 
intfoperxiefit  aroctei1  to  the  piatfom  specie 
wpt&wefTt&tion 


^derive* 


*iationa7e& 

@y  standardizing  the  way  infoimatien 
flows  between  tampontnis  and 
simulations,  simulation  developers  wifi 
need  less  integration  code  to  translate 
data  and  requests 


“Tx, 


lication  will 
ent  of  any 
mponents. 


info  rm  ation 


simulations,  simulation  developers  will 
need  less  integration  code  to  translate 
data  and  requests 


A  A  A  A 


(ttoffr  Core  Requirements) 
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Requirements 
Traceability:  TEAMS  Core  Values 


QfpjCEOFNAMJLRES^Qy 


?EAM* 


Standard  Interfaces 

notes 

TEAMS  will  prowo1*?  interfaces 

for  &Aivteti on  etemente  as  sspfsifbm 

catrtpQn&nts,  Any  tfe  t&top&r  of  erfb&r  systeiir 
eiMimng  TEAMS-Complrance  to  any  tfegf&e 
1 4v/l  use  these  interfaces  as  the  external 
boundary  of  thw  i&plementetiOrt 


Platform  Independence 

notes 

The  interfaces  that  TEAMS  provides  will  not 
oreclude  any  particular  implementation 
olatfbrm  in  their  design.  This  includes  but  is 
not  limited  to  considerations  suet?  as 
transport  meciranism.  operating?  system,  and 
orogramming  language. 

Standard  Interfaces 

notes 

TEAMS  will  provide  standardized  interfaces 
for  simulation  elements  as  well  as  platform 
components.  Any  developer  of  either  system 
c/aiming?  TEAMS-Compltance  to  any  degree 
will  use  these  interfaces  as  the  external 
boundary  of  their  implementation . 

Open  Standards 

notes 

To  promote  the  usefulness  of  standardized 
interfaces.  TEAMS  must  make  those 
interfaces  public  and  available  to  any 
interested  party. 

Model  Realizable  Systems 

notes 

The  interfaces  that  appear  in  the  TEAMS 
model  wilt  reflect  actual  systems  in  the  real 
►vortd.  7?ris  includes  designed  systems  as 
well  as  physical  constraints  placed  by  the 
environment. 


Extensible  Interfaces 

notes 

7??e  TEAMS  interfaces  will  not  be  binding 
contracts  o7  behavior,  but  rather  a  basis  of 
communication  between  components  These 
interfaces  will  be  extendable  to  include  new 
wrays  of  communication,  and  new  jbetraviors 
of  estajb/isf?ed  communications. 


Model  Realizable  Systems 

notes 

TTro  interfaces  that  appear  in  the  TE  AMS 
model  will  reflect  actual  systems  in  the  real 
wortd.  This  includes  designed  systems  as 
tvell  as  physical  constraints  placed  foythe 
en  w/onwent 


Evolving  Standards 

notes 

TEAMS  will  update  their  model  periodically 
whenever  such  ci?ang?es  ate  required  to 
oreserve  an  up-to-<iate  reflection  of  actual 
systems. 

Loosely  Coupled  Interfaces 

notes 

TEAMS  will  design  the  interfaces  such  that 
they  do  not  depend  on  the  internal  structure 
of  any  other  interface  and,  where  possible , 
do  not  depend  on  the  existence  of  another 
interface  at  all . 

Support  Different  Levels  of  Detail 

notes 

Mflrere  possibie,  TEAMS  will  use  value  types 
ajbstractiy  to  avoid  specifying  the  level  of 
detail  present. 

Standard  PSM  Implementation  Strategies 

notes 

7o  promote  interchange  of  implemented 
components.  TEAMS  will  provide  several 
oopular  Implementation  strategies.  77?is  will 
ore  vent  any  loss  of  communication  due  to 
different  mappings  from  the  TEAMS  platfom 
independent  modei  to  the  platform  specif/c 
implementation . 

Tiers  of  Interfaces 

notes 

77?e  interface  modei  will  layer  its  interfaces 
such  that  higher  levels  completely  compose 
lower  levels,  and  no  interface  or  behavior 
everdepends  on  the  lower  structure  of  an 
interface  in  its  own  tier.  Communication 
between  components  will  only  exist  within 
the  same  interface  tier,  or  when 
communicating  with  the  component's  parent 
or  child  component. 

Sponsor  Requirements 
Mapped  to  TEAMS  Core  Values 


ncevCE  OF  NAWL  . 


TEAMS  Perspective: 

SysML  Pros 


ncevCE  OF  NAWL  . 


Pros 

Requirements 

-  Explicitly  lay  out  requirements  and  consequences 

Views  and  Viewpoints 

-  Can  separate  requirements  and  model  views  based 
on  stakeholders  concerns 

Structure 


-  Ability  for  model  structure  to  verify  requirements 

•  Can  search  for  requirements  that  aren’t  verified 

•  Can  search  for  model  components  that  aren’t  justified 

-  Separation  of  structure  from  behavior 

•  SysML  BDDs  vs.  IBDs  and  Activities  allow  for  clear  separation 

•  UML  allows  this,  but  easier  to  implement  in  SysML 

•  Behavior 


-  Dashed  line  for  activity  flow  is  more  aesthetically  pleasing 
•  vs.  UML  solid  line 
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TEAMS 

Stakeholder  Requirements 


req  Requirements  [Requirements] 


y 


«view» 

Sponsor  Requi  re  ments 

mr^i  +  Contractor  Interoperability 

nm  +  Less  Component  Integration  Time 

mm  +  Model  Realizable  Systems 

nm  +  Reduced  Duplicate  Efforts 

firm  +  Reuse  Legacy  and  Hew  Components 

rrrii  +  Room  for  Future  Growth 


«view» 

Si  mulati  on  Developer  Requi  re  ments 

firm  +  Continue  using  Legacy  Systems 
firm  +  Design  Flexibility 
nm  +  Easier  Maintenance  and  Upgrades 
firm  +  Less  Component  Integration  Time 
rum  +  Simulate  Real  Situations 


«view» 

Co  mponent  Devel  oper  Requi  re  ments 

im  +  Design  Flexibility 
nm  +  Develop  New  Approaches 
nn  +  Intellectual  Property  Rights 
nm  +  Goffpoffefff  drfegrafroff  firne 
mm  +  Model  Real  Components 
nm  +  Scalable  Component  Design 
nm  +  Simulation  Interoperability 


«view» 

Fleet  Requi  re  ments 

nm  +  Better  Systems 
nm  +  Commonality 
nm  +  Highly  Detailed  Simulations 
nm  +  Shorter  Acquisition  Period 


TEAMS  Perspective: 

SysML  Pros 


ncevCE  OF  NAWL  . 


Pros 


•  Requirements 

-  Explicitly  lay  out  requirements  and  consequences 

•  Views  and  Viewpoints 

-  Can  separate  requirements  and  model  views  based  on  stakeholders  concerns 

•  Structure 

-  Ability  for  model  structure  to  verify  requirements 

•  Can  search  for  requirements  that  aren’t  verified 

•  Can  search  for  model  components  that  aren’t  justified 

-  Separation  of  structure  from  behavior 

•  SysML  BDDs  vs.  IBDs  and  Activities  allow  for  clear  separation 

•  UML  allows  this,  but  easier  to  implement  in  SysML 

•  Behavior 

-  Dashed  line  for  activity  flow  is  more  aesthetically  pleasing 

•  vs.  UML  solid  line 
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Torpedo 
Block  Definition  Diagram 


QfpCEOFNAMJL^S^Qy 


k  bifrdc» 

S&nsorSysfem 


0.x 

^sensQts 


15 — S' 


+/s*ftwrsj 


A  0,." 


ftblddta 

Sensor  Syste  m : :  Sensor  Control  I  er 


0..1 


^block^ 

Sensor  System:: 
StindardDstectionSy 


+  crossings 


nd 


+£J0S$i 


r>gg\/ri 


ablddtB 

Sensor  System :: 

Thrasbdd  Crossing  Cottactor 


0.1 


^blocks 

Sensor  System:: 

Post  Detadi  on  PFOtas 


+:p05t/V  1 


«characteristicAgent» 
Agents:  Wake  Model 

«characteristicAgent» 
Agents::SelfNoise  Model 

«characteristicAgent» 
Agents: :  Radi  atedNoi  se  Model 

«characteristicAgent» 
Agents::Collisi  on  Model 
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MotionSystem::Propulsion 
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Torpedo  Internal 
Block  Definition  Diagram 


ncevCE  OF  NA^  . 


^EAM® 


«interface» 


+  Am {time Step  :Fme*)  :  iyoid 
+  DisamQ  :  void 

+  Gt?eckFuzes$ime  Step  :  Time  *)  :  \foid 


z 


warheads  [□..*] 

interface 


d  StandardTorpedo  [StandardTorpedo] 


«interface» 

CommunicafionCfrannei 


+  LtstenForfVfessagesQ  :  iyoid 
+  tfe c e i: i i-e fife ssa g e  ft m e Step  :7 me*)  :  Octet  Stmam 
+  Sendf\fessage{h?sg  .Octet  Stfeam ,  time  Step  .Tme*)  . 
+  StopListeningQ  :  vo rtf 
+  Resume  pme  Step  :7fme*)  :  void 


void 


«block» 

StandardTorpedo 


ransmitfcarams  .TransAcousticParams)  :  void 

leceive(params  :RecvAcousticPacams)  :  Multi  ChannelDiscrete  Signal 
iancelReceive  (signal  Multi  CtiannelDiscrete  Signal)  :  void 
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ReceiveRequest. 
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interface  u  J 

Amt  {time  Step  Time*)  :  void 
Disarm  (fi  :  void 

CheckFuzesftime  Step  .  rime*)  :  void 


+  ListenForMessagesQ  :  void 

+  Recei  ve/Vfessage  pme  Step  :  Firne  *)  :  Octet Stieam 
+  SendMessage  (tosg  :Octet Stream,  time  Step  :F/me*)  :  void 
+  StopListeningO  :  void 
+  Resume  (time  Step  :Ftme*)  :  void 
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Navigate  (bmd  .GuidanceCommand ,  time  Step  :  Time  *)  :  void 
Resume  (time  Step  .rime*) :  void 
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+  Navigate  {bmd  .Guidance  Commend „  time  Step  .Tme*)  :  void 
+  Resume  {time  Step  :7ime*)  :  iyoid 
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interface 


guideCmd 
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interface 
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Sense pmd  :SensorParameters.  time  Step  .  rime*)  :  i 
Resume  (time  Step  .  rime*)  :  void 


1-5^ 

interface  body  [1] 

motion  wake  collision  self  radiated  highlight 

— E - E - E - E — HE - HE — 


E - 1E - HE — HE — ^-E - HE 


Navigate  params  :Na vigationParameters)  :  void 


radical  pme  Step  :  rime)  :  void 
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iki£tii*k  nJecut* 


Torpedo  Sensor 
Activity  Diagram 


f  \Gt  OF  NAVAL  RES&V?^ 

^oRPEDOENTEfiPRi^ 
^rtrrrFfook1 1 1  ™" 

fjJ/fi 10 1101 0011^^ v 

ADVANCED^  )? 
Vx  ^JaiiHCVaiMw* — r^>  A.y/tVJ 

NG  &  SIMUlW^! 


Senselnitial 


This  activity  shows  a  Standard 
Detection  System  executing  a  Sense 
command.  This  means  the  tactical 
controller  decided  to  execute  a 
detection  cycle  on  this  Sensor  System 
during  this  time  step.  The  Resume 
activity  is  nearly  identical  to  this, 
except  initially  control  is  transferred  to 
the  subsystem  that  is  in  progress,  and 
there  is  no  new  incoming  command 
(control. Interpret  gets  the  old  one  from 
its  static  storage). 


Block 


Undersea  World 
Definition  Diagram 


Simulation 
Internal  Block  Definition 


“World” 

Diagram 


^^^PEDOENTEfiPfiil^^; 

r  ^TTmnnPS^* 

II^iCADVANCEOjlT, 


^d&M© 


World 


dnterfaoe» 

+  fiic^&7 1)  :  T^'ec-iory  ■tffH&fy} 

+  n-ufit  :  .Vjfirai1  ery£ 

+  e^Vo^o/r  Gc  0  .-Jboo^ejr? 

+  Galculab&Motiot?  e Step  :  voicf 

+  Ge- q V u  s  (bTrr  :  Lefrgtfr 

+  GefDej&/iaffeJef(}  :  D&^/isfie^d 
+  ^gf?fr£?J7i:(fr7dfe-x  jWafarrai'j  :  WgWigfrt 

+  Jrste oiliUo/-se  (f niervs J  drr  : 


World 
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Acoustic  Properties _ 

Internal  Block  Definition  Diagram 


ibd  Acoustic  Properties  [AcousticProperties] 


y 


«interface» 

A  cousticProperfies 


+  tmuzmissiousO  :  Acoustic  TmusmissionDatabase  faver/} 
+  scatter^  :  Volume Scattering  Strength  fa  very} 

+  speed  0  :  Sound  SpeedDateGrid  fauety} 

+  j£jo undenesQ  :  AcousttcBoundaryHfauery} 


7T 


«  b  I  o  ck» 

AcousticPropei 


0—Oun 


transmissions 


^interface* 

A  caustic  Transmission 


+  n urn  Transmissions^  :  Natural  fat 
+  Transmission  {index  .Natural)  :  A 
+  Store  {iequest  .Acoustic  Transmit 


scatter  [1] 


£2 


^interface* 
Volumes  caterings 


+  Scattering  Strength  {incoming  .'Direction,  outgoing  :Dii 


ibd  AcousticProperties  [AcousticProperties] 


y 


ainterface* 

A  cousticProperties 


transmissions^  :  Acoustic  TransmissionDatabase  finery} 
scatter/)  :  Volume  Scattering  Strength  fiuery} 
speed  0  :  Sound  Speed  Data  Grid  finery} 
boundaries/)  :  AcousticBoundary/]  fiuery} 


ablock* 

Acousti  c  Properti  es 


o — n 


3 


transmissions  [1] 


« interface* 

A  coustio  TransmissionDatabase 


num  Transmissions/)  :  Natural  fiuery} 

Transmission /Index  Natural)  :  AcousticTransmitRequest 
Store  /iequest  .Acoustic  Transmitflequest)  :  void 


=£ 


« interface* 

VolumeScatteringSfrength 


Scattering  Strength /incoming  .Direction,  outgoing  .Direction,  freq  frequency,  pos  Position)  .Real 


ainterface* 

SoundSpeedDafaGrid 


south  West  Comer/)  :  Horizontal  Position  fiuery} 
north EastComerO  :  HorizontalPosition  finery} 
north  SouthResolutionO  -  Natural  finery} 
eastWestResolution  /)  :  Natural  finery} 

Location  (north  Index  Natural,  eastihdex  .Natural)  :  PtorizontalPosition 
Profile  (north  index  Natural,  east/ndex  Natural)  :  Sound  SpeedPtoftle 


ainterface* 

A  cousficBoundary 


Forward  Reflection  pos  .HorizontalPosition .  freq  .Frequency,  grazing  .Angle)  :  Complex 

Bistatic  Scattering  Strength  incoming  .Direction,  outgoing  .£Vrecfrorr,  freq  .Frequency,  pos  :HorizontalPosition)  :  Real 
ScatteringOensity/pos  .HorizontalPosition .  input : Angle,  output  .Angle)  :  Real 
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TEAMS  Perspective: 

SysML  Pros 


Pros 


Requirements 

-  Explicitly  lay  out  requirements  and  consequences 

Views  and  Viewpoints 

-  Can  separate  requirements  and  model  views  based  on  stakeholders  concerns 

Structure 

-  Ability  for  model  structure  to  verify  requirements 

•  Can  search  for  requirements  that  aren’t  verified 

•  Can  search  for  model  components  that  aren’t  justified 

-  Separation  of  structure  from  behavior 

•  SysML  BDDs  vs.  IBDs  and  Activities  allow  for  clear  separation 

•  UML  allows  this,  but  easier  to  implement  in  SysML 

Behavior 


-  Dashed  line  for  activity  flow  is  more  aesthetically 
pleasing 
•  vs.  UML  solid  line 


ncevCE  OF  NAWL  . 


tc 


Simulation 

World”  Activity  Diagram  1*:f|fgf 


R  u  n  S  i  m  u  I  ati  o  n  I  n  iii  a  I 


X 


Derive  Ti me  Step 

timeStep 


X 


9 


+  continue  ! 


^parallel  s- 


Platform 


^allocate# 


platforms  :  Platform  Model 


■X 


Tactical 


timeStep 


X 


f  Calculate  Motion 
-|~fr  |  time  Step 


OO 


r 


Get  Debris  Field 


RunSimulationlnitia 


Report  Results 

"\ 

X; _ 

J 

I 


RunSimulationFina 


Run  Si  mulation 


Platform  Extrinsics 


platforms  :  Platform  Model 


Cal  cul  ate  Moti  on 
|  time  Step 


Get  Debris  Field 


Platform  Intrinsics 


platforms  :  Platform  Model 


I  Execute  AgentsTi  me  Cycle 
X  time  Step 


39 


Solid  Line  Representation 


^aiNG&SIMUlff^ 

^EAM® 


Run  Simulation  Initial 


± 


Derive  Tim*  Step 

tiiritStep  I 


lb 


continue 


■■"  cparallels> 


^Ull  OQ 


platforms  :PI? 


Tacli 


timgStep 

v___ 


1 


I  Calculate 
-MtimeStep 


SelDebri 


R  u  n  S  i  m  u  I  ati  o  n  I  n  iti  a  I 


nr 


Report  Results 

V _ 

J 

v 

® 

RunSimulationFir 


RunSi  mulatio 


Platform  Extrinsic: 


V 

nmesiep  | ^  | 

w  y 

i^i 11 

platforms  : PI atf or m Model 


Cal  cul  ate  Moti  on 
— ^  |  time  Step 


Get  Debris  Field 


platforms  :  Platform  Model 


Execute  AgentsTi  me  Cycle 
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TEAMS  Perspective: 

SysML  Cons 


loRPEDO  ENTERpftac 


ADVANCED. 


I6&  SIMULA^ 


C  7  : - 


Cons 


Allocating  CIM  to  PIM 

-  Difficulty  with  abstract  activities 

•  Exit  path  dependent  on  logic  within  an  activity  is  not  accessible 
and  can’t  be  modeled 

•  Not  represented  well  in  either  UML  or  SysML  -  tactical 
controller  example 

Implementing  PIM 

-  Not  “direct”  for  some  SysML  features 

•  Flow  ports,  continuous  activities,  parametric  constraints  involve  more  components  than  just 
themselves 

•  Flows  in  “real  systems”  easier  to  represent 

•  Flows  in  software  modeling  are  open  to  interpretation 

-  Requires  additional  documentation  of  model  to  bridge  between  SysML  feature  and 
executable  code 


TEAMS 
Tactical  Controller  Example 


ADVANCED 
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ADVANCED. 


ING&  SIMULA^ 


c  7  : - 


TEAMS  Perspective: 

SysML  Cons 


Cons 


Allocating  CIM  to  PIM 

-  Difficulty  with  abstract  activities 

•  Exit  path  dependent  on  logic  within  an  activity  is  not  accessible  and  can’t  be 
modeled 

•  Not  represented  well  in  either  UML  or  SysML  -  tactical  controller  example 

Implementing  PIM 

-  Not  “direct”  for  some  SysML  features 

•  Flow  ports,  continuous  activities,  parametric  constraints 
involve  more  components  than  just  themselves 

•  Flows  in  “real  systems”  easier  to  represent  than  simulations 

•  Flows  in  software  modeling  are  open  to  interpretation 

-  Requires  additional  documentation  of  model  to  bridge 
between  SysML  feature  and  executable  code 


Lessons  Learned  (ifsfs^fzyz 
and  Value  Added  ^lilfr 

•  Requirements  traceability  is  vital  to  the  success  of 
several  TEAMS  projects 

-  ONR  TEAMS  standard  framework  and  interfaces 

-  OSD-ATL  feasibility  study 

-  TOGAF/MDA  Synergy  Project 

•  SysML  was  designed  with  “real”  systems  in  mind 

-  where  UML  is  software  oriented 

•  Perceived  concreteness  -  simulated  vs.  actual  system 

-  not  just  one  way  to  design  interfaces,  need  recommendations 
for  implementation 

•  Still  need  some  UML  features  not  present  in  SysML 

-  «lnstantiate»  or  «create»  for  dynamic  allocation 

•  Still  need  guidance  on  how  to  best  implement 
parametrics  and  constraints  for  modeling  and 
simulation 


^cevCE  OF  NAVAj-  R£S£a&s>,  . 

r  ^TtTmnnPS^* 
^1011010011^^3 
!1  JlJ^iCADVANCEDjSr 


IG&  SIMULA^ 


OMG  SE  DSIG  Recommendation 


“Clarify  the  distinction  between  the  domain 
model  and  the  simulation  design  model.  ” 


*Reference  SE  DSIG  minutes  from  OMG  San  Diego  Meeting  on  March  27,  2007 
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incose  Integrating  SysML  Models 

with  Simulation  Models 


SYST 

MODE! 

LANGU 


•  Goal 


-  Integrate  system  design  models  with  simulation  and  analysis 
models 

•  Use  SysML  models  to  specify  an  executable  architecture 

•  Use  simulation  and  analysis  models  to  analyze  performance 

•  How  can  they  work  together  ? 

-  Plug  the  SysML  executable  architecture  model  into  a  simulation 
infrastructure  to  establish  a  dynamic  interface 

-  Use  the  executable  architecture  model  to  control  the  sequence 
of  activities  (e.g.  detect  target,  launch  weapon) 

-  Use  the  simulation  model  to  compute  the  parameter  values  (e.g. 
missile  range  to  target  vs.  time) 

•  What  is  needed? 

-  Approach  to  use  SysML  architectural  model  to  specify  simulation 
requirements  (use  of  parametrics?) 


-  Harmonization  between  SysML  and  simulation  standards  (i.e. 
HLA)  ? 


Source:  Sanford  Friedenthal,  Lockheed  Martin,  OMG  SE  DSIG  Chair  -  Recommendation  to  TEAMS  Project 


Future  Direction 

Working  to  Establish  an  Activity  for  SysML  / 

Simulation  Integration  Approach 

-  Formulation/establishment  during  INCOSE  MBSE 
Workshop  in  Albuquerque  on  January  24-25 

-  Liaison  to  the  INCOSE  Model  Base  Systems 
Engineering  (MBSE)  Initiative 

-  Keep  abreast  of  industry  related  activities 

-  Help  to  foster  interaction  in  this  area  across 
industry,  government  and  academia  to  help  move 
towards  the  INCOSE  MBSE  Vision. 

-  Explore  this  integration  through  SISO. 


fEAM* 
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SE  for  S&T 


Applications  of  System  Engineering 
to  Pre-Milestone  A  Projects 

Lori  F  Zipes 

NSWC-PC  Panama  City  FL 


[Agenda 

■  What  does  “Pre-Milestone  A”  mean? 

■  Can  you  do  SE  at  this  point? 

■  Why  should  you  do  SE  at  this  point? 

■  How  do  you  do  SE  at  this  point? 


Pre-Milestone  A 


A  DoD  term  that  captures  the  concept 
development  and  concept  refinement  stages 

INCOSE  (handbook/15288)  equivalent  is 
“Concept”  stage 

Often  known  in  industry  as  “Study  Period” 


[jCIDS  “wall  chart” 


Zoom  to  Pre  MS  A 


Concept  Refinement  Phase 


Retine  initial  concept  Develop  Teclntology 
Development  Sttategy 


Joint  Capabilities  integration  fi  Development  System  ■  Acronyms 


CDD- Capability  Envelopment  Docum  ent 
CPD- Capability  Production  Document 
OCR  -  EOTFrlLPF  Change  Recomm  endation 
DOTH  LPF-  Doctrine,  Organization,  Training. 

Materiel,  Leader ihlp,  Penonnel,  and  Faclllflei 
FOC-  Full  Operational  Capability 
CD-  hltlal  Capabilities  Document 
CC-  hltia  I  Operational  Capability 


JCD- Joint  Capability  i  Docum  ent 
J  ROC  -  Joint  Requlrem  enti  Oversight 
Council 

MUA-  Military  Utility  An ei  im  ent 
NCCVv'-RM  -  Met  Centric  Cperatiom 
Warfare-  ReTerence  Model 
KIP- Key  hterface  Fictile n 
KPP-  Key  Perform  ance  Parameter 


Family  of  Joint  Future  Concepts 
Concept  of  Operations 
Automated  Standards  Profile 
Capability  Roadmap 


ot 


l!  Draft  ! 
CDD 


DraftCDD  tr  i  iowi  t>  lllistrafc 
lie  Import*  ice  ofai  earfyitart 
oh  titr  critical  doci  me  it  All 
decline  it  are  1 1  mated  as  draft 
attie  worklig  level. 


Pre  MS  A  engineering  aspects 


INPUTS 

(ICD  ^ 

•AoA  Ft  an 
•Exit  Criteria 

•Alternative  Maintenance 
Logistics  Concepts  j 


OUTPUTS 


/’’•Frelirn  Syr,  Spec 
•T&E  Strategy 
SEP 


\ 


Support  &  Maintenance 
Concepts  &  Technologies 
•inputs  to: 

-draft  CDD 


-AoA 


■T°S 

V_  -CostIMan power  Est. 


[zoom  to  before  Pre  MS  A 


Concept  Development 


* 


Joint 

Capabilities 
Integration  & 
Development 
System 

(need  driven) 


Decision  Points  Milestones 


Functional 
Area  Analysis 

I 


Strategic  Guidance 
I 

Family  of  Joint  Future  Conceits 
Concepts  of  0|>erations 
Joint  Tasks 

,--i77P - : - 


Functional 

Heeds 

Analysis 


1--1V  if 


X 


l'- 


■  r  „ 


JCD 


Integrated 

Architectures 


Ideas  for 
Materiel 
Approaches 


Ideas  for 
Non-materiel 
Approaches 

■DOTMLPF 

Mnaly  c.» 


Functionaf  Sofution 
Analysis 


Analysis  of 
Materiel/ 
Hon- Materiel 
Approaches 

ApproachT"! 

ApproaclTil 


Can  you  do  SE  at  this  point? 

■  Not  a  lot  of  people  do.... 


Can  you  do  SE  at  this  point? 

■  Not  a  lot  of  people  do.... 


....but,  YES! 


Why  should  you  do  SE  at  this  points 


■  Typical  Pre-  MS  A  situation: 

a.  A  new  technology  looking  for  a  problem 

b.  A  problem  with  a  potential  solution 
concept  that  leverages  one  or  more 
technologies.  There  is  a  vision  that 
needs  some  detailing  and  proving. 


Why  should  you  do  SE  at  this  points 

■  Q.  So  why  can’t  you  just  “play  around” 
with  the  ideas  &  technologies  until  you 
get  them  robust  enough  to  warrant 
“real”  system  engineering  activities? 

■  A.  You  can,  but  you  might  miss  out  on 
some  important  things. 


Common  Vision 


■  Do  all  team  members  understand  the 
end  goal? 

■  Is  there  a  documented  “big  picture” 
technical  approach? 


A  notional  example 


■  New  concept  to  use  underwater 
ultrasound  to  measure  ship  hull 
thickness 

o  Transducer  mfr 
o  COTS  scope 

o  Cables  &  misc  COTS/custom  as  needed 


■  If  the  transducer  mfr  is  not  aware  this 
will  be  diver-held,  he  may  make 
something  with  great  resolution,  but 
unmanageably  large. 

o  He  needs  the  big  vision  so  he  can  make 
proper  development  tradeoffs,  even 
within  his  own  “sandbox” 


If  you  don’t  do  good  requirements 
development,  you  might  find  out  too 
late  that  divers  have  certain  racks  or 
cases  all  their  equipment  fits  into,  and 
the  scope  you  chose  does  not  fit! 


If  you  don’t  talk  to  a  logistician,  you 
won’t  know  that  there  is  a  hull  cleaning 
system  that  you  might  easily  attach  to 
or  integrate  with,  making  your  concept 
much  more  attractive  to  users. 


[how  to  do  SE  at  this  point 

■  Things  to  focus  on: 

o  System  Engineering  Plan  to  include: 
o  Requirements  Development 
o  Configuration  Management 
o  Risk  Management 
o  Quality  Assurance 


■  Follow  the  OSD  guidance:  Systems 
Engineering  Plan  Preparation  Guide 

http://www.acg.osd.mil/se/publications.htm 

Section  3  will  guide  you  through. 

The  plan  need  not  be  burdensome. 

Evolve  it  as  you  progress. 


Requirement  Development 


■  Identify  and  document  the  source  of 
your  requirements,  then  determine  if 
you  really  have  them  all! 

■  How  did  you,  or  will  you,  validate  them 
(who  are  your  users  and 
stakeholders?) 

■  How  will  you  manage  them? 

■  Do  you  need  to,  or  want  to  “ architect ” 
your  system  ? 


I  Configuration  Management  J 

■  Identify  what  types  of  info  require  CM 

o  SEP,  Proj  Plan,  Work  Packages,  Reqmts 
o  Final  Reports,  Design  Baselines 

■  Plan  for  version  control  (1  .x  or  date  or...) 

■  Define  a  review/change/approval  process 

■  Where  will  they  be  stored? 


[sample  CM  content 


Document  Title 

Project  Plan 

Systems  Engineering  Plan 
User  Requirements  Doc 


Author  Signature  Authority 

PM  Sponsor 

Lead  Systems  Eng.  PM 
Systems  Engineer  Sponsor 


This  list  will  be  updated  as  needed  during  execution  of  the  project.  Documents  subject  to 
formal  configuration  control  will  be  required  to  pass  through  peer  review  and  signature  for 
both  initial  generation  and  any  subsequent  revisions.  Formality  and  breadth  of  the  peer 
review  will  be  at  the  discretion  of  the  document  author  with  concurrence  from  the 
signature  authority.  All  formal  configuration  controlled  documents  will  include  version  and 
date  information  on  the  title  page,  and  a  revision  history  page.  Version  numbers  shall  be 
O.X  until  first  signature  approval.  Thereafter  minor  revisions  shall  be  numbered  by 
iterating  the  numeral  to  the  right  of  the  decimal,  major  revisions  shall  increment  the 
number  to  the  left  of  the  decimal. 

The  Project  Manager  shall  be  responsible  for  maintaining  accurate  knowledge  of  and 
access  to  the  most  current  version  of  each  formally  controlled  project  document. 

General  configuration  control  for  all  working  level  documents  will  be  executed  via  the  use 
of  a  collaborative  data  site.  Users  will  be  responsible  for  maintaining  current  versions  of 
documents  they  post  for  team  use.  The  team  will  utilize  the  site  as  the  primary  source  for 
working  level  information,  to  minimize  version  control  issues  created  by  the  e-mailing  or 
other  uncontrolled  distribution  of  documents. 

As  this  project  advances,  a  more  robust  configuration  management  process  may  be 
required,  to  include  specific  tools.  This  section  will  be  updated  accordingly  at  that  time. 


Risk  Management 


■  Consider:  schedule,  cost,  resources, 

technical 

■  Suggestion:  PM  identifies  schedule, 
cost  &  resource  risks.  SMEs  identify 
technical  risks. 

■  Plan  for  assessment,  mitigation... 

■  Suggestion:  review  risks  at  team 
meetings 


[Risk  Information  Form  sample 

Risk  Information  Form 


Risk  Identification  Number  |  |  Date  Entered: 

Risk  Title: 

Priority: 

Statement  of  Risk 

Description  of  risk:  | 

Causes:  | 

1 

Relationship  to  other  risks: 

1 

Probability  of  Occurrence: 

Consequence: 

Time  Sensitivity  (when  mqiht  risk  occur): 

Risk  Handling  Plans:  | 

Status  Information:  ~f 


Risk  Information  Form 


Risk  Information  Form 


Risk  Identification  Number  |SNS RI-001  |Date  Entered:  06  June  06 

Risk  Title: 

Un-approved  requirements 

Priority: 

Medium 

Statement  of  Risk 

Description  of  risk:  | 

I/prioritized  by  the  user  community.  Several  remain  poorly  defined: 
cgraphic  location  for  environmental  specifics. 
r  importance:  very  small  mine  (<6”)  neutralization  /  success  rates, 

Requirements  remain  unapproved 
“in  stride”  breaching  capability,  gei 
Some  are  of  questionable/uncleai 
command  detonation  requirement 

Causes:  [ 

Broad  search  was  done  to  gather  requirements,  some  findings  conflict,  some  are  lacking  verifiable 
source,  have  not  yet  had  the  opportunity  to  vet  with  operational  community  to  resolve  these  issues 

Relationship  to  other  risks:  | 

1 

it  decision  making,  particularly  during  tradeoff  efforts 

Uncertainty  of  requirements  impac 

Probability  of  Occurrence: 

B 

Unlikely 

Consequence: 

D 

Major  Impact 

Time  Sensitivity  (when  mgiht  risk  occur): 

4Q  2006 

Risk  Handling  Plans:  | 

ith  operational  community.  If  no  validation  meetings  have  been 
i  sponsor  of  situation. 

Continue  effort  to  make  contact  wi 
scheduled  by  15  Sept  2006,  inforn 

Status  Information:  ~  j 


09/11/06  Contact  made  with  Route  Clearance  Training  school  at  Ft  Leonard  Wood,  MO  (Army); 
tentative  requirements  validation  meeting  week  of  10  Oct  2006.  Contact  made  with  MCES  Lejeune 
NC  (USMC);  tentative  requirements  validation  meeting  week  of  27  Sept  2006. 


Quality  Assurance 


■  Suggestion:  Set  basic  standards  for 
things  like  meeting  agendas,  minutes, 
action  items  &  follow  up 

Don’t  make  them  burdensome,  but  have 
some  simple  expectations  &  make 
sure  they  are  followed. 


Other  thoughts... 


■  Strongly  recommend  periodic  team 
meetings,  particularly  if  team  is 
geographically  dispersed. 

■  Don’t  make  any  of  the  “process”  work 
unnecessarily  burdensome.  Use 
common  sense,  follow  SE  principles. 


[parting  Thought 


Plan  your  dive; 


dive  your  plan 


Questions,  comments? 


Program  Management  vs 
Systems  Engineering 


How  different  are  they? 

Lori  F  Zipes 
NSWC-PC 
Panama  City,  FL 


Overview 


o  PMBoK  review 

o  DAU  Guidebook  review 

o  INCOSE  handbook  review  (15288) 

o  What  are  the  PM's  goals,  the  SE's 
goals? 

o  What  should  a  PM  do,  what  should 
an  SE  do? 

o  PM  skills,  SE  skills 
o  Can  one  person  do  both? 


Perspective  for  this  presentation 


o  DoD 

o  Technical  Programs  (heavy  SE  role) 
o  Possible  R&D  bias  (mine) 


PMBoK  (3rd  Ed  2004) 


o  44  "Project  Management  Processes" 

o  Each  is  associated  with  one  of  5 
"Project  Process  Groups" 

Initiating,  Planning,  Executing,  Monitoring,  Controlling 

o  Each  is  also  associated  with  one  of 
9  "Knowledge  Areas" 

Integration,  Scope,  Time,  Cost,  Quality,  Human  Resource, 
Communications,  Risk  and  Procurement  Management 

Let's  look  at  those  44  processes... 

...very  quickly 


KA  4.  Project  Integration  Management 


o  4.1  Develop  Project  Charter 

o  4.2  Develop  Preliminary  Project 
Scope  Statement 

o  4.3  Develop  Project  Management 
Plan 

o  4.4  Direct  and  Manage  Project 
execution 

o  4.5  Monitor  and  Control  Project  Work 
o  4.6  Integrated  Change  Control 
o  4.7  Close  Project 


KA  5.  Project  Scope  Management 


o  5.1  Scope  Planning 
o  5.2  Scope  Definition 
o  5.3  Create  WBS 
o  5.4  Scope  Verification 
o  5.5  Scope  Control 


KA  6.  Project  Time  Management 


o  6.1  Activity  Definition 
o  6.2  Activity  Sequencing 
o  6.3  Activity  Resource  Estimating 
o  6.4  Activity  Duration  Estimating 
o  6.5  Schedule  Development 
o  6.6  Schedule  Control 


KA  7.  Project  Cost  Management 


o  7.1  Cost  Estimating 
o  7.2  Cost  Budgeting 
o  7.3  Cost  Control 


KA  8.  Project  Quality  Management 


o  8.1  Quality  Planning 
o  8.2  Perform  Quality  Assurance 
o  8.3  Perform  Quality  Control 


KA  9.  Project  Human  Resource 
Management 

o  9.1  Human  Resource  Planning 
o  9.2  Acquire  Project  Team 
o  9.3  Develop  Project  Team 
o  9.4  Manage  Project  Team 


KA  10.  Project  Communications 
Management 

o  10.1  Communications  Planning 
o  10.2  Information  Distribution 
o  10.3  Performance  Reporting 
o  10.4  Manage  Stakeholders 


KA  1 1 .  Project  Risk  Management 


o  11.1  Risk  Management  Planning 
o  11.2  Risk  Identification 
o  11.3  Qualitative  Risk  Analysis 
o  11.4  Quantitative  Risk  Analysis 
o  11.5  Risk  Response  Planning 
o  11.6  Risk  Monitoring  and  Control 


KA  12.  Project  Procurement  Management 


o  12.1  Plan  Purchases  and  Acquisitions 
o  12.2  Plan  Contracting 
o  12.3  Request  Seller  Responses 
o  12.4  Select  Sellers 
o  12.5  Contract  Administration 
o  12.6  Contract  Closure 


DAU  Defense  Acquisition  Guidebook 


o  Designed  to  compliment  DoDD 
5000.1  and  DoDI  5000.2  "by 
providing  the  acquisition  workforce 
with  discretionary  best  practice..." 

a  how-to  guide 

o  Program  Management  (DoD  style)  is 
throughout  the  document 

o  Chapter  4  is  Systems  Engineering  in 
specific  ...so  we'll  look  at  that  a  bit 


DAU  Guidebook  Ch  4  -  SE 


o  Technical  Management  Processes: 

•  Decision  Analysis 

•  Technical  Planning 

•  Technical  Assessment 

•  Requirements  Management 

•  Risk  Management 

•  Configuration  Management 

•  Technical  Data  Management 

•  Interface  Management 

some  of  these  look  familiar... 


DAU  Guidebook  Ch  4  -  SE 


o  Technical  Processes: 

•  Requirements  Development 

•  Logical  Analysis 

•  Design  Solution 

•  Implementation 

•  Integration 

•  Verification 

•  Validation 

•  Transition 


DAU  Guidebook  Ch  4  -  SE 


o  Also  mentioned: 

•  Quality 

•  Master  Plan  /  Schedule 


these  ring  a  bell  also... 


INCOSE  SE  Handbook  V3 


o  Technical  Processes  (Ch  4) 

o  Project  Processes  (Ch  5) 

o  Enterprise  and  Agreement 
Processes  (Ch  6) 


Consistent  with  ISO/IEC  15288 


INCOSE  SE  Handbook  V3 


o  Technical  Processes 

•  Stakeholder  Requirements  Definition 

•  Requirements  Analysis 
Architectural  Design 


Implementation 

Integration 

Verification 

Transition 

Validation 

Operation 

Maintenance 

Disposal 


very  similar  to  DAU  Guide 
technical  processes 


INCOSE  SE  Handbook  V3 


o  Project  Processes 

•  Project  Planning 

•  Project  Assessment 

•  Project  Control 

•  Decision  Making 

•  Risk  and  Opportunity  Management 

•  Configuration  Management 

•  Information  Management 

quite  similar  to  DAU  Guide 
technical  management  processes , 
which  were  similar  to  PMBoK 


INCOSE  SE  Handbook  V3 


o  Enterprise  and  Agreement  Processes 

•  Enterprise  Environment  Management 

•  Investment  Management 

•  System  Life  Cycle  Process  Management 

•  Resource  Management 

•  Quality  Management 

•  Acquisition 

•  Supply 


a  few  more  familiar  terms... 


PMBoK  vs  DAU  vs  INCOSE  Hdbk 


So  who  does  what? 


PMBoK 


DAU 


4.1Develop  Project  Charter 

Technical  Planning 

4.2  Develop  Preliminary  Project 
Scope  Statement 

Technical  Planning 

4.3  Develop  Project  Management 
Plan 

Technical  Planning 

4.4  Direct  and  Manage  Project 
execution 

Decision  Analysis 

4.5  Monitor  and  Control  Project 
Work 

Technical  Assessment 

4.6  Integrated  Change  Control 

Configuration  Mgmt,  Tech 
Data  Mgmt 

4.7  Close  Project 

5.1  Scope  Planning 

Technical  Planning 

5.2  Scope  Definition 

Technical  Planning 

5.3  Create  WBS 

Technical  Planning 

5.4  Scope  Verification 

Technical  Assessment 

5.5  Scope  Control 

Decision  Analysis, 
Technical  Assessment 

INCOSE 


Project  Planning,  SLC  Process  Mgmt,  Investment  Mgmt 

Project  Planning,  SLC  Process  Mgmt 

Project  Planning,  Resource  Mgmt,  Investment  Mgmt 

Project  Assessment,  Project  Control 

Project  Assessment,  Project  Control,  Decision  making 

Project  Assessment,  Project  Control,  Configuration  Mgmt 

Project  Control 

Project  Planning,  Enterprise  Environment  Mgmt,  SLC 
Process  Mgmt 

Project  Planning 

Project  Planning 

Project  Assessment,  Enterprise  Environment  Mgmt 


Project  Control 


PMBoK 


DAU 


6. 1  Activity  Definition 

Technical  Planning 

6.2  Activity  Sequencing 

Technical  Planning 

6.3  Activity  Resource  Estimating 

Technical  Planning 

6.4  Activity  Duration  Estimating 

Technical  Planning 

6.5  Schedule  Development 

Technical  Planning 

6.6  Schedule  Control 

Technical  Assessment 

7. 1  Cost  Estimating 

Technical  Planning 

7.2  Cost  Budgeting 

Technical  Planning 

7.3  Cost  Control 

Technical  Planning 

8.1  Quality  Planning 

Technical  Planning 

8.2  Perform  Quality  Assurance 

Quality 

8.3  Perform  Quality  Control 

Quality 

INCOSE 


Project  Planning 

Project  Planning,  Decision  Making 
Project  Planning,  Resource  Mgmt 
Project  Planning 
Project  Planning 

Project  Control,  Decision  making 
Project  Planning 

Project  Planning,  Resource  Mgmt 

Project  Control,  Decision  making,  Resource  Mgmt 

Project  Planning,  Quality  Mgmt 

Configuration  Mgmt,  Quality  Mgmt 

Project  Control,  Quality  Mgmt 


PMBoK 


DAU 


9. 1  Human  Resource  Planning 

9.2  Acquire  Project  Team 

9.3  Develop  Project  Team 

9.4  Manage  Project  Team 

10.1  Communications  Planning 

10.2  Information  Distribution 

10.3  Performance  Reporting 


Technical  Planning 


Tech  Data  Mgmt 
Tech  Data  Mgmt 
Tech  Data  Mgmt 


10.4  Manage  Stakeholders 

11.1  Risk  Management  Planning 

Technical  Planning,  Risk 
Mgmt 

11.2  Risk  Identification 

Risk  Mgmt 

1 1.3  Qualitative  Risk  Analysis 

Risk  Mgmt 

11.4  Quantitative  Risk  Analysis 

Risk  Mgmt 

1 1.5  Risk  Response  Planning 

Technical  Planning,  Risk 
Mgmt 

1 1.6  Risk  Monitoring  and  Control 

Risk  Mgmt 

INCOSE 


Project  Planning,  Enterprise  Environment  Mgmt,  Resource 
Enterprise  Environment  Mgmt,  Resource  Mgmt 
Resource  Mgmt 

Project  Control,  Resource  Mgmt 
Project  Planning,  Information  mgmt 
Information  mgmt 
Information  mgmt 

Enterprise  Environment  Mgmt 

Project  Planning,  Risk  and  Opportunity  Mgmt 
Risk  and  Opportunity  Mgmt 

Project  Assessment,  Risk  and  Opportunity  Mgmt,  Decision 
making 

Project  Assessment,  Risk  and  Opportunity  Mgmt,  Decision 
making 

Project  Planning,  Risk  and  Opportunity  Mgmt,  Resource 
Mgmt 

Project  Assessment,  Risk  and  Opportunity  Mgmt 


PMBoK 


DAU 


12.1  Plan  Purchases  and 
Acquisitions 

12.2  Plan  Contracting 

12.3  Request  Seller  Responses 

12.4  Select  Sellers 


12.5  Contract  Administration 


Technical  Planning 
Technical  Planning 


12.6  Contract  Closure 


INCOSE 


Project  Planning,  Acqusition  &  Supply  Processes 
Project  Planning,  Acqusition  &  Supply  Processes 


Acqusition  &  Supply  Processes 

Project  Control,  Decision  making,  Acqusition  & 
Supply  Processes 

Project  Control,  Acqusition  &  Supply  Processes, 
Resource  Mgmt 


Acqusition  &  Supply  Processes 


PMBoK 


DAU 


INCOSE 


Requirements  Development 

Stakeholder  Requirements  Definition 

Logical  Analysis 

Requirements  Analysis 

Design  Solution 

Architectural  Design 

Implementation 

Implementation 

Integration 

Integration 

Verification 

Verification 

Validation 

Validation 

Transition 

Transition 

Operation 

Maintenance 

Disposal 


PMBoK  vs  DAU  vs  INCOSE  Hdbk 


So  (again)  who  does  what? 


PM  vs  SE:  what  are  their  goals? 


o  PM  is  accountable  for  the  success  of 
the  entire  program  and  all  aspects 
of  it. 

o  SE  is  responsible  for  the  technical 
success  of  the  program. 


Some  “clear”  distinctions 


These  are  "owned"  by  the  PM: 

Enterprise  Environment  Management 

Investment  Management 

System  Life  Cycle  Process  Management 


Some  “clear”  distinctions 


These  are  "owned"  by  the  SE: 

Stakeholder  Requirements  Definition 

Requirements  Analysis 

Architectural  Design 

Implementation 

Integration 

Verification 

Validation 

Transition 

Operation 

Maintenance 

Disposal 


Some  “not  so  clear”  distinctions 


These  are  probably  "owned"  by  the  PM,  but 
require  inputs  and  assistance  from  the  SE: 

Project  Planning 
Project  Assessment 
Project  Control 
Decision  Making 
Risk  and  Opportunity 
Management 

Configuration  Management 
Information  Management 
Resource  Management 
Quality  Management 
Acquisition 
Supply 


Getting  the  Right  People 


o  What  makes  a  good  PM? 


o  What  makes  a  good 


A  “good”  PM  -  the  Program  Leader 


o  Is  ideally  a  business  or  management 
major,  or  has  a  strong  background  & 
skills  in  these  areas 

o  Beware  the  Technical  major  as  PM! 

•  Might  get  stuck  "in  the  weeds,"  lack  program 
level  vision. 

•  Tend  to  micromanage  technical  aspects. 

•  Might  get  focused  on  technical  problem  and 
not  make  the  best  programmatic  decision. 

•  May  not  have  the  discipline  to  manage 
rigorously  (think  CMMI:  do  "coders"  like  CMMI?) 


A  “good”  SE  -  the  Technical  Leader 


o  Is  (hopefully!)  a  technical  major 

o  Beware  the  Non-technical  major  who  has 
some  sort  of  SE  role  (or  if  there  is  no  SE) 

•  May  lack  ability  to  form  and  propagate  an 
overarching  technical  vision 

•  Might  be  more  of  a  manager  than  a  leader 

•  Might  not  have  the  proper  knowledge  to 
resolve  technical  conflict  or  make/approve 
technical  decisions. 


PM  vs  SE  perspectives 


It  is  not  necessarily  bad  for  there  to 
be  a  bit  of  friction  between  the  two 

...because  sometimes  the  optimal 
technical  solution  is  not  the  optimal 
programmatic  solution 


So,  can  one  person  do  both? 


o  On  a  "small"  program 

o  Very  early  in  a  program  (even  a  big 
program) 

o  On  a  non-complex  program 

•  No  hardware/software  mix,  single 
technology,  few  or  no  external 
interfaces... 


Things  to  watch  out  for  in  these  cases 


o  Need  to  get  an  individual  with 
strong  and  broad  technical 
knowledge  and  management  skills 

o  Make  sure  they  have  a  mental 
concept  of  their  two  "hats"  and 
when  they  need  to  wear  each  one. 


Perspective  -  a  parting  thought 


o  Both  people  need  to  appreciate  the 
role  of  the  other  person,  determine 
mutually  agreeable  dividing  lines  for 
their  responsibilities. 


Questions,  comments? 


A. 

THE  UNIVERSITY 
of  Arizona. 


Raytheon 

Customer  Success  is  Our  Mission 


Autonomic  GIG  Management  & 
Security  Agent  Technology 


32M1 


10th  Annual 

NDIA  System  Engineering  Conference 
October  22-25,  2007 


Don  P.  Cox,  Missile  Systems 
Youssif  Al-Nashif,  University  of  AZ 
Salim  Hariri,  PhD,  University  of  AZ 
(520)794-8186)  dcox@raytheon.com 
Abstract  #  5386 


I  L 


NON-ITAR 


Agenda 


Raytheon 


-The  GIG 
■  Autonomia 


Attack  Detection  &  Defense 


Conclusions 


THE  UNIVERSITY 

of  Arizona. 


NON-ITAR 


10/24/2007  I  Page  2 


Raytheon 


Thank  you  ! 
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Introduction 


Circa  2000  -  F-18 


Preflight  status  awareness 
Tactical  view  integrated  manually 
Update  via  voice 
Limited  data  security 
Radar  flight  following 


Circa  2015  -F-22 

•  Integrated  Global  Information  Grid 

•  Real-time  data  from  forward  C4I  center 

•  Dynamic  (In-flight)  situation  updates 

•  Secure  data-link  (Intrusion  aware) 

•  C2  AC  mission  capability  awareness 


ik 
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GIG  History 

■  The  Clinger-Cohen  Act,  1996 

-  Information  Technology  Management  Reform  Act 


■  DoDCIO  Memorandum  “Global  Information  Grid,”  (9/99) 

-  Version  1.0  Approved  by  DoD  CIO  --  8/01 

-  Version  2.0  Approval  by  DoD  CIO  --  8/03 


DoD  Directive  Number  8100.1  (11/03) 

-  Global  Information  Grid  (GIG)  Overarchin 
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information  Management 
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GIG  Architecture  (“Beer  Barrel”) 


Warfighters  (Joint  Services) 


Common  set  of  information  capabilities 

•  GIG  Enterprise  Services  (GES) 

•  Core  Enterprise  Services  (CES) 

•  Communities-of-lnterest  (COI) 

•  Service  Oriented  Arch  tecture  (SOA) 


IT  Infrastructure 

DoD  Foundation 

•  Policy/Doctrine/Governance 

•  Standards/Engineering/Architecture 
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GIG  Security  Challenges 
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Autonomic  Computing 


Self-Protecting 


Detect  internal/external  attacks  and  protect  it’s  resources 
from  exploitation. 


I  I  I 


. 


i  i  i 


Self-Optimizing  Detect  sub-optimal  behaviors  and  intelligently  optimize 

resource  performance. 


-r 


H 


a 


4T 


Self-Healing 


Detect  hardware/software  failures  and  reconfigure  to  permit 
continued  operations. 

N1WAI 


KU 

Self-Configuring  Dynamically  change  resource  configuration  to  maintain 

system  &  application  requirements. 

LY\  I  I  I  IHTI  I  I  I  IVkff  ~ 
/In  I  II  NT  MM  tty 


JL 

THE  UNIVERSITY 
or  Arizona. 


(I 


Autonomia  will  ultimately  provide  all  necessary  tools  for 
control  and  management  of  GIG  networks  and  services. 
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Autonomia  Classification 


Raytheon 


■  Policy  rule  -  Condition-action  policy  dictates  the  actions  that 
should  be  taken  whenever  the  system  is  in  a  given  state. 


■  Optimization  -  Analytical  techniques  are  used  to  model  the 
overall  system  behavior  and  services  through  a  utility  function  that 
is  used  to  select  the  optimal  adaptation  strategy. 


■  Artificial  Intelligence  -  Al  planning  &  learning  techniques 
model  system  behavior  by  using  data  mining  and  statistical 
techniques. 


• 

1 

y 

H5 

r 
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Autonomia  Architecture 


System  Development  Environment 
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Autonomic  Agent  Approach 


CMI  (CCMI)  j 


,ni  jj 


Planning 


Managed 

Environment 


knowledge 

Repository 


Execution 


Compound  CRM  (CCRM) 
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Autonomia  Testbed 
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Test  Results 


USAF  testing  of 


Autonomia 


(Detect 


ion) 


290,870  Netflow  records  -  (70K  normal  +  220K  abnormal) 


.rh-fWK 


Attack  Category 


z 

25* 

_ 

_ 

1  rHl 

Attack  Methods 


Results 


Scanning 


Xprobe2,  APNET,  Nikto,  Traceroute,  Nessus, 
SARA,  NMAP  ,  Queso 


Whisker,  enum 


Detected 


Not  detected 


Passive  Scanning 


Ettercap 


Not  detected 


Exploits 


Ownstation,  Snooqer,  SMB/RPC  Nuke,  Jolt2, 
RPC  DCOM,  Octopus,  Killthemessenger 


Not  detected 


R2L 


Netcat 


Detected 


DoS  Attack 


TCP  SYN  Flooding  Attack,  UDP  flooding, 
ICMP  flooding 


Detected 


Worm 


theodin  worm 


Detected 


ml 
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Feature 


Validation 


USAF  LAN  (capture) 

-  DARPA  Dataset  KDD99  (Lincoln  Labs) 

-  9  Weeks  raw  TCP  dump  data. 

-  5M  connection  records  +  49K  training  records 

-  41  features 

-  22  different  attack  types 


JL 
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Class 

UA  Approach 

Winner  Entry 
using  C5.0 

CTree 

Normal 

98.45% 

99.5% 

92.78% 

Dos 

99.93% 

97.1% 

98.91% 

U2R 

92.55% 

88.13% 

R2L 

^■§2.46% 

8.4% 

7.41% 

PROBE 

99.91% 

83.3% 

50.35% 

NON-ITAR 
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■  Autonomia  framework  -  autonomic  computing  systems  and  applications 

■  Supports  “design-in”  or  legacy  resources  and  software  systems 

■  Initial  Autonomia  software  modules  to  focus  on  self-protection  (minimal) 

■  Existing  Experimental  Testbed  (University  of  Arizona,  Tucson) 

■  Effective  in  detecting  and  protecting  the  networks  but  immature 

■  Wide  range  of  network  attacks 

■  High  detection  rate  accuracy  +  very  low  false  alarms 

■  Limits: 

-  Could  not  detect  attacks  that  require  payload  monitoring  or  analysis 

-  Internal  or  insider  attacks  (network  monitors  or  ‘bad  eggs’) 

JL . 
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Raytheon 


Network  Attack  Technology 


Viruses:  Computer  program  which  distributes  copies  of 


without 


ission  or  knowledge  of  the  user. 

Z®1 


hat  reproduce  and  run  jpdependently,  and 
;ross  network  connections. 


Trojans:  Impostor  files  that  claim 
but,  in  fact,  are  malicious. 


Others: 


-  Protocol  (TCP)  attacks 

K 
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Additional  Research 
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—  Virtual  Network  Models 
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GIG  SCOPE 

Requirements  and 
Architecture 


Policy 


Programs  & 
Experimentation 


“Develop,  maintain  and  facilitate  the  implementation  of  a  sound  and  integrated 
information  technology  architecture  for  the  executive  agency.” 

(40  U.S.C.  Section  1425) 


i» 


THE.  UNIVERSITY 

or  Arizona. 


Copyright  ©  2007  Raytheon  ComDanVjAnUnoublished  Work.  All  rights  reserved. 


NON-ITAR 


10/24/2007  1  Page  23 


Raytheon 


NON-ITAR 


Self-Protection  Engine 


Module 
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Primary  goals:  1)  Detect  network  attacks,  known  or  unknown, 

2)  Proactively  prevent  or  minimize  impact  on  network  operations  and  services.  / 


Onlin 
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Integration  of  isolated  solutions 


JL 
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Cost  Effective 


Consumer 

Systems 


Fault  Tolerant 


Financial  Systems 


High  Performance 

Scientific  Computing 
Systems 


Current 

Integrated  System 

High  Performance, 
Fault  Tolerant 
Security  and 
Cost-Effective 
Systems 


1 

ft 

A 

> 
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\Autonomie  Component 


Cost-Effective 

Consumer 

Systems 


Secure 

Military 

Systems 


High 

Fault-Tolerant  Performance 
Financial  Scientific 

Systems  Computing 

Systems 


System  Management  Editor 


Raytheon 


System  Development  Environment 
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Management  Web  Services 
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Compound  CRM  (CCRM) 


Raytheon 


Planning 


Manages  Cpmpoun|d  Components 

-Analy 

■ 
ii 


CCMI  Ports 

1.  c. 


lanning 


3.  Operation 


-Execution 

TTTJ  Lb 


/"Autonomic  Compound  Component 

C Compound  CRM  (CCRM) 


Compound  i 

cm  (ccmi)! 


cm\ 


Analysis 


Managed 

Environment 


knowledge 

Repository 


Execution 


Monitoring 
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Managed  GIG  Environment 


Larger  autonomic  systems 

-  Hierarchical  manner 

-  Composed  of  many  autonomic  compound  components 

-  Deployed  dynamically 

-  Once  deployed,  becomes  self-maintaining  (“living”) 


M, 
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NetFlow  Data 


A. 

THE  UNIVERSITY 
of  Arizona. 


Variable 

Hid 

Bytes 


Pkts 


lnput snmp 
Output snmp 
src addr 
dst addr 
Prot 

L4 src/dst port 
Next hop 
Src/dst AS 
Src/dst mask 
TcpJIags 
Src  tos 


Definition 

Sequence  id 

Number  of  bytes  in  this 
interval  for  a  connection 

Number  of  packets  in  this 
interval  for  a  connection 

related  incoming/outgoing 
interface  information 


IP  source  and  destination 
address  information 


Protocol  number 
Layer  4  port  information 
Next  hop  information 
Srouce/destination  AS 
Mask  of  the  src/dst  IP 
Bitwise  OR  of  tcp  flags 
TOS  of  the  connection 
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FEATURE  X 

l(X;  DOS) 

l(X;DOS) 

/  H(DOS) 

count 

0.647571 

0.899405 

dst bytes 

0.512438 

0.71  1719 

d  st h  ost s  am  es  rc po  rt rate 

0.382541 

0.531308 

srv_count 

0.338744 

0.470478 

dst_host_count 

0.308133 

0.427963 

src_bytes 

0.290684 

0.403728 

dst_host_srv_d  iff_host_rate 

0.274275 

0.380937 

dst_h  ost_s  rv_co  u  nt 

0.165472 

0.229823 

s  rv_d  if  f_host_rate 

0.165142 

0.229364 

dst  host  same  srv  rate 

0.149499 

0.207638 

dst  host  diff  srv  rate 

0.14109 

0.195959 

diff_srv_rate 

0.084967 

0.1  18009 

dst_h  ost_s  rv_ser  ro  r_rate 

0.081939 

0.1  13804 

s  am  e  s  rv_r ate 

0.080769 

0.1  12179 

dst_host_serror_rate 

0.076816 

0.106688 
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Defining  the  future 


I  nteg rated  Risk  and 
Earned  Value 
Management 


2007  NDIA  Systems  Engineering  Conference 
San  Diego,  CA 


October  24,  2007 

Paul  Davis 

Northrop  Grumman  Corporation 


Contents 


Uncertainty  management  premises 
State  of  industry 
“As-ls”  risk  management 
Baseline  planning 
Risk  identification 
Monitoring  and  control 
Integration  approaches 
Summary 


Uncertainty  Management  Premises 


A  failure  to  meet  project  objectives  is  a  failure  in 
uncertainty  management 

Uncertainty  management 

■  Risk  management  (RM)  -  minimizing 
negative  consequences 

■  Opportunity  management  -  maximizing 
positive  consequences 

Risk  management  = 

Uncertainty  management 

Uncertainty  management 

■  Affects  project  execution 

■  Changes  the  project  future  by 

■  Identifying  uncertainty 

■  Measuring  uncertainty 

■  Risk  exposure  (likelihood  X  impact) 

■  Improving  effective  of  uncertainty  handling 

■  Improving  decision  making  to  successfully  achieve  objectives 


■  Improved  decision  making  a  key  focus 


NORTHROP  GRUMMAN 


State  of  I  ndustry 

■  NDIA  -  Program  Management 
Systems  Committee  Survey* 

■  121  respondents 

■  Study  findings: 

■  RM  and  EVM  have  separate 
process  owners  76%  of  the  time 

■  System  engineering 

■  Program  management 

■  Project  control 

■  Business/financial  management 

■  Risk  management  seldom  predicts  near-term  issues 

■  Majority  (70%)  strongly  believes  in  the  value  of  integrated  RM  and  EVM 
even  though  only  34%  said  they  were  successfully  integrating  them 


■  RM  and  EVM  integration 

■  Oct  2003  to  Jun  2004 


“Failure  to  integrate  RM,  cost- 
risk  analysis,  and  EVM 
contributes  to  overruns.  The 
program  manager  is  denied 
clear  visibility  of  quantitative  RM 
that  could  increase  the 
probability  of  mission  success.” 

Peter  Teets,  former  Under 

Secretary  of  the  Air  Force 


*  “Integrating  Risk  Management  with  Earned  Value  Management”,  at 
www.ndia.org/Content/ContentGroups/Divisions1/Procurement/ 


NORTHROP  GRUMMAN 


Typical  “As- 1  s"  Risk  Management 
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As- 1  s"  Risk  Management  Process 
Capability  -  Cost  Control 

Past  Performance  Contract  References 

Variable:  CPI  Inverse 


Anderson-Darling  Normality  Test 


I  I  l  I  l  I  l 

0.8  0.9  1.0  1.1  1.2  1.3  1.4 

I  I  I  I  I  I  I 


95%  Confidence  Interval  for  Mu 


A-Squared: 

0.252 

P-Value: 

0.664 

Mean 

1.10210 

StDev 

0.14044 

Variance 

1 .97E-02 

Skewness 

0.206006 

Kurtosis 

0.821268 

N 

11 

Minimum 

0.85000 

1st  Quartile 

1 .00000 

Median 

1.12000 

3rd  Quartile 

1.18600 

Maximum 

1 .38000 

95%  Confidence  Interval  for  Median 


95%  Confidence  Interval  for  Mu 
1.00776  1.19645 

95%  Confidence  Interval  for  Sigma 
0.09813  0.24646 

95%  Confidence  Interval  for  Median 
1.00000  1.18880 


2.  Label  a  Normal  curve 

-  Average 

-  Standard  deviation 

-  USL  (and  shade  to  LEFT  for  Area  1 ) 

-  LSL  (and  shade  to  LEFT  for  Area  2) 


Average  costs  exceed 
budgeted  costs  by  10.7% 


Less  than  25%  of 
projects  meet 
budgeted  costs 


Xbar 

s 

USL 

LSL 

1.102 

0.14 

1 

0.77  sigma 

- USL 

- LSL 

s 

0.7  0.8  0.9  1.0  1.1 


1.2  1.3  1.4  1.5  1.6  CPI’1 
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Project  Baseline  Planning  I  ntegrating  Risk 
Management 


Plan  for  Risk 
Management 


Scope 

Definition 

Direct  &  Manage  Project 
Execution 

* 

Develop  PM 
Plan  ^ 

Integrated  Change 

 Control 

Monitor  and  Control 
Project  Work 


Plans  that  include  risk  handling  with  accomplishment  criteria 
Risk  identification  that  is  comprehensive  and  decision  based 
Risk  handling  activities/actions  in  project  schedules  and  logs 
Scheduled  risk  management  monitoring  and  control  reviews 


Identify 

Analyze 

Handle 

Risk 

Risk 

Risk 

i 


Effective  Risk  Management  Actions  are  Comprehensive 
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A  Structured  Risk  and  Opportunity  I  dentification  (SROI ) 
Approach  Is  Effective  in  Identifying  More  Uncertainties 


Comparison  of  risk  counts  from  uniformly  distributed  risks  over  a  (5  X  5) 
likelihood-by-impact  linear  risk  space  with  average  counts  from  6  SRI  pilot  projects 


16.00 

14.00 

12.00 

£  10.00 

3 

<5  8.00 

V ) 

6.00 

4.00 

2.00 

0.00 


Conclusions  from  analyzing  risks  from  the  six  SRI  pilot  projects: 

-  Project  risk  identification  identifies  most  high  exposure  risks  (1 6-25) 

-  Low  exposure  risks  (1-  5)  remain  unidentified  even  with  SRI 


ILn 


V. 


10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25 

Risk  Exposure  J  V _ J 


Y 


Unidentified  risks 

•  50%  of  risks 

•  20%  of  risk  exposure 


SRI  identified  risks 

•  42%  of  risks 

•  55%  of  risk  exposure 


Project-identified  risks 

•  8%  of  risks 

•  25%  of  risk  exposure 
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I  nteg rating  Risk  Monitoring  and  Control  with  Project 
Monitoring  and  Control 


Scope 

Definition 


Develop  PM 
Plan 


Risk 

Management 

Planning 


i 


Direct  &  Manage  Project 
Execution 


Monitor  and  Control 
Project  Work 


Integrated  Change 
Control 


Quantifies  cost,  schedule,  and  technical  risk  exposure 
Risk  exposures  included  in  cost/schedule  forecasts 
Estimates  at  complete  (EAC)  computed  with  the 
addition  of  risk  and  opportunity  exposures 
Risk  management  tracked  by  EAC,  CPI  and  SPI 


r 


Risk 

Qualitative  & 

Risk 

Risk 

Identification 

- ► 

Quantitative 

- ► 

Response 

— ► 

Monitoring 

Risk  Analysis 

Planning  1 

and  Control 

Integrated  RM  &  EVM  assists  decision  making 
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RM  and  EVM  I  ntegration  Approaches 


Barriers  to  risk  management  integration* 

■  Contractual  incentives  ■  Organizational 

■  Technology  -  tools  ■  Baseline  instability 

■  RM  or  EVM  process  maturity  ■  Emotional  u 

■  Internal/external  management  cultures  -1,! 

o  1 

RM-EVM  integration  approaches  £ 

■  EAC  with  and  without  risk  exposure  ° 

«0.6 

■  Residual  uncertainties  in  forecasts  « 

with  statistical  profiles  and  EAC  ellipses  f" 

■  Risk  handling  earned  value  monitoring  -  “u 
residual  risks  monitored  against  plans  c 

■  Cost  and  schedule  performance  indices 
(CPI  and  SPI)  monitoring  and  control 


■  Focus  on  risk  handling,  not  mechanics 

*  “Integrating  Risk  Management  with  Earned  Value  Management”,  NDIA  Study  Report  2005 
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"Earned  Value"  Monitoring  Measures  Risk  Handling 
Effectiveness 


■  Monitors  actual  handling 
performance  against  plans 


■  Performance-based  earned 
value®measures 

■  A  means  to  measure 
uncertainty  management 
effectiveness  performance 


■  Measures  effectiveness  of 
uncertainty  management, 
not  just  task  completion 


■  Triggers  uncertainty 
management  corrective  actions 


Performance-Based  Earned  Value  is  registered  with  the  U.S.  Patent  and 
Trademark  Office  by  Paul  Solomon. 
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Cost/  Schedule  Performance  Monitoring  Provides 
Leading  I  ndicators  for  Corrective  Action 


Risk  Management  Process  Effectiveness  Monitoring 


/VOHTIMtiOH  C?iF«L«WfA¥ 


Summary 


RM-EVM  integration  provides  leading  indicators  that 
increase  response  time  and  probability  of  success 


A  structured  risk  identification 
approach  increases  risk 
assessment  comprehension 


Quantified  uncertainty  metrics  are 
a  basis  for  effective  management 


Alternative  RM-EVM  integration 
approaches  can  be  selected  to  meet  project  needs 


■  Focus  on  uncertainty  handling  and  project  decision 
making  --  not  on  uncertainty  computation  mechanics 
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POSTGRDUATE 

SCHOOL 


Quantitative  Comparison  of 
Alternative  Designs  for  a  Joint  C4I 
Capability  Certification 
Management  (JC3M)  System 


A  Student  Project 


Gregory  A.  Miller 
Naval  Postgraduate  School 
Monterey,  CA 


Ian  Finn 

Marine  Corps  Tactical 
Systems  Support  Activity 
Camp  Pendleton,  CA 


Outline 


□  Introduction  &  motivation 
□A  tailored  SE  process 

□  Problem  refinement 

□  Design  Alternatives 

□  Modeling  &  Simulation 

□  Life  Cycle  Cost  Estimates 
□Analysis  of  Alternatives 

□  Conclusions  and  further  study 


JC3M  -  Paper  5407 
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Motivation: 


Acquisition  system  &  SoS  integration  needs 


Step  #1 
Develop  Each 
System  in 
Isolation 


Army  System  X 
Marine  System  Y 
Air  Force  System  A 
Navy  System  N 
Marine  System  Z 


Developers 
&  Program 
Offices 


Step  #2 
Perform 
Developmental 
Testing  on 
Each  System 


Testing 

Agencies 

Step  #3 
Integration 
Testing  ( w/o 
SoS  rqmnts) 


►  Joint  C4I 
System  of 


Systems 


Operating 

Forces 


Fielding  Decisioi 
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Current  SoS  Testing  and  Fielding 


□  Problem  with  SoS  Testing 

■  No  Performance  Measurements 

■  What  Architecture  is  Appropriate?  Joint 
C4I  SoS  are  Large  and  Constantly 
Changing 

■  Testing  Every  SoS  Function  is  Impossible 


■  Hard  to  Determine  What  Failure  is  Since 
Quality  of  Service  Requirements  Change 


JC3M  -  Paper  5407 
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What  is  the  Real  Problem? 
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What's  the  Solution? 


Develop  a  System  that  articulates  SoS  capabilities, 

iA/hi A+hi ai*  aaaI^  CaC  AAmnAnAnt  A\/n+Am  m  iaaak+0 


Elicit  Requirements 
Define  Performance 


Define  Architecture 


ID  Systems 


Paper  review 

(On  paper,  did  each 
SoS  component  meet 
articulated  SoS 
capabilities?) 


Formal  Report 
Certification 


Live  Testing 

▼ 


Simulation 


JC3M  -  Paper  5407 


6 


JC3M  in  Testing  and  Fielding 


(Currently 

unavoidable) 


(Currently 

unavoidable) 


(Replaces  current  SoS 
testing  methodology) 


□  JC3M  goals: 

■  Acquire  objective  SoS  Performance 
Measurements  for  Acquisition  and  User 
Communities 

■  Produce  Decision  Data  for  Stakeholders 

■  Provide  confidence  in  SoS  Performance 
for  Users 


JC3M  -  Paper  5407 


Systems  Engineering  Process 
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Revised  Problem  Statement  0^ 


□  Original  problem  focus 
■  Define  Threshold  Values 


□Research  ife^ealed  the 
true  proble 


m 


□Refined  problem  focus: 

■  Define  Measures  to  be 
Evaluated 


Problem 

Refinement 


\ 


Re-Evaluate 
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Revised  Problem 
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JC3M  Value  Hierarchy 


□  Developed  from  Refined  Problem 
Statement 

□Based  on  Stakeholder  Analysis 


Functional 


JC3M  -  Paper  5407 


Non-Functional 
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Plan  C4I  SoS  Evaluation 


JC3M  Functional  Decomposition 


JC3M  -  Paper  5407 
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Define  Evaluation  Criteria  1.3  0'^P? 
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JC3M  Value  Hierarchy 


EZ 
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Evaluation  Measures 


Percentage  of 
Traceable  Measures 

Days  to 

Plan  Evaluation 

Quality  of  Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

JC3M 

Define  Measures 

Planning  Results 

Planning  Results 

Input  System 

Input  System 

Function 

1.3.2 

1.4.3 

1.4.3 

Flexibility 

Flexibility 

4.1 

4.1 

Definition 

Alternative  generated 

Elapsed  time  (in 

Quantify  the  overall 

Divide  percent  change 

Divide  percent  change 

measures,  traceable 

days)  of  planning  for 

quality  of  the 

in  labor  hours  to 

in  duration  to  conduct 

to  stakeholder 

C4I  SoS  evaluation 

planning  documents 

conduct  planning 

planning  phase  of 

requirements,  divided 

produced. 

phase  of  JC3M  by  the 

JC3M  by  the  percent 

by  the  number  of 

percent  change  in 

change  in  systems 

measures  generated 

systems  under  test. 

under  test. 

by  the  alternative. 

(Quantifies  ability  to 

(Quantifies  ability  to 

scale.) 

scale.) 

Ratio  level  data, 

Ratio  level  data  >  0 

Ordinal  -  Low, 

from  0  -  100% 

hours 

Medium,  High 

Ratio  level  data  from 

Ratio  level  data  from 

0  —  oo 

0  —  oo 

Rationale  and 

Identifies  objectivity 

Predicts  SoS 

Identifies  predicted 

Predicts  changes  in 

Predicts  changes  in 

Relevance 

of  performance 

evaluations  that  can 

utility  of  alternative. 

cost  of  SoS  evaluation 

duration  of  SoS 

measures. 

be  conducted  in  a 

based  on  size. 

evaluation  based  on 

year. 

Quality  of  the 

size. 

Performance 

planning  products 

Can  be  used  to 

measures  traceable  to 

Alternatives  that 

drives  the  overall 

determine  most 

Can  be  used  to 

doctrinal  references 

permit  multiple  SoS 

value  of  the 

effective  alternative 

determine  most 

will  be  perceived  as 

evaluations  generate 

alternative. 

based  on  SoS  size. 

effective  alternative 

objective,  increasing 

data  to  support 

based  on  SoS  size. 

the  value  of  the 

fielding  decisions 

evaluation. 

sooner. 

Alternatives 
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Morphological  Box  Process  0 


Define  the 
Problem 

ID 

Systems 

Umiec-Xest 

Define 

Criteria 

Ensure 

Readiness 

Have  SMEs 
Do  It 

/What  PM\ 

Av  Requests  J 

Ask  Users 

/  PM 

V  Review  J 

Acquisition 
Manager 
Defines  / 

[  Document 
Review 

/What  PM\ 

\  Asks  For  J 

SAK~Review 

/AcquisitiorK/ 

(  Manager  J 
X^Def  i  ne$/ 

Engineering 

Document 

Review 

Test 

Everything 

Test  Manager 
Review 

Get  from 
CDD 

Ask  JITC 

Stakeholder 
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Alternative  #1 


J1EL 

ft,  f  Avg  v 


"System  Capabilities  Review  (SCR)" 
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Alternative  #2 


I  ,17V 
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Differences 


SCR  Alternative 
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JC3M  Fu 

FCB  Alternative 


Alternatives  Summary 


Personnel 

Use 

Scope 

Measures 

FEDOS 

Internal 

Past 

Service  test 

Stakeholder 

aareement 

MC3T 

Internal  + 

External 

Proof  of 
concept 

Service  system 

Doctrine 
developers  & 
stakeholders 

JTEM 

CTM 

Internal 

Model 

Joint  Mission 
Effectiveness 

Assessment 

Doctrine,  System 
documentation 

SCR 

Internal 

Proposed 

Joint  capability 
assessment 

Doctrine,  System 
documentation 

FCB 

Internal  + 
External 

Proposed 

Joint  capabilitv 
assessment 

C4I  SME 

oanel 
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Fill  in  the  blanks! 


© 


SCR 


Percentage 

of 

T  raceable 
Measures 

Days 
to  Plan 
Evaluation 

Quality 

of 

Planning 

Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

EM  #1: 
1.3.2 

EM  #2: 
1.4.3 

EM  #3: 
1.4.3 

EM  #4: 
4.1 

EM  #5: 
4.1 

FEDOS 

MC3T 

JTEM  -  CTM 

A 

A 

A 

k 

FCB 

A— 

JL 

/ 

— d 

L- 

Offline  / 
Evaluation 


POW-ER 


Offline 
Evaluation' 


Arena 
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M&S  Overview 


JC3M 


K 


■ 
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M&S  Results 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

1.3.2 

1.4.3 

1.4.3 

4.1 

4.1 

FEDOS 

140  days 

0.87 

0.86 

MC3T 

121  days 

0.78 

0.78 

JTEM  CTM 

73  days 

1.04 

0.83 

FCB 

158  days 

0.97 

0.97 

SCR 

127  days 

0.71 

0.71 
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Complete  EM 


Percentage 

Traceable 

Measures 

Days  to  Plan 
Evaluation 

Planning 

Output 

Quality 

Labor 

Elasticity 

Duration 

Elasticity 

% 

Days 

Likert  Scale 
1-4 

Unitless 

Unitless 

Ideal  Value 

100% 

Less  is  better 

4  is  Ideal 

Less  is 
better 

Less  is 
better 

FEDOS 

0 

140 

3.17 

0.87 

0.87 

MC3T 

72 

121 

3.25 

0.78 

0.78 

JTEM  CTM 

92 

73 

3.42 

1.04 

0.83 

SCR 

92 

158 

3.00 

0.98 

0.98 

FCB 

88 

127 

2.75 

0.72 

0.72 
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LCCE  -  Cost  Breakdown  Structure 


\s 


S7 


JC3M  -  Paper  5407 


26 


Life  Cycle  Phases  of  JC3M 
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LCCE  -  Cost  Summary 


Alternatives 

Life-Cycle  Year 

Total  Cost 

($> 

1 

2 

3 

4.. .9 

10 

FEDOS 

1,052,527 

419,497 

419,497 

419,497 

52,200 

5,010,706 

MC3T 

1,169,414 

525,537 

525,537 

525,537 

52,200 

5,975,913 

JTEM-CTM 

1,030,000 

2,470,000 

1,169,414 

558,535 

52,200 

6,972,824 

FCB 

2,323,117 

650,223 

650,223 

650,223 

52,200 

8,127,101 

SCR 

2,121,421 

624,451 

624,451 

624,451 

52,200 

7,719,232 

Interpretation:  The  delta  between  the  highest  and  lowest  LCCE  ^  $3M, 
which  is  not  a  significant  sum  over  a  ten  year  span. 
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Value  Modeling  Overview 


Percentage  of 
Traceable  Measures 

(%) 

Days  to  Plan 
Evaluation 

(Days) 

Qualit 

1 

0 

Ideal  Value 

100% 

Less  is  better 

4 

FEDOS 

0 

140 

MC3T 

72 

121 

JTEM  CTM 

92 

73 

SCR 

88 

127 

FCB 

92 

158  / 

EM 

Weights 


Percentage 

of 

Traceable 

Measures 


0.248 


- v" 

Days  to 

Plan 

Evaluation 


0.058 


Translation  of  raw 
measurements  into  a 
normalized  set  of  weighted 
values  that  can  be  added. 


Quality  of 
Planning 
Outputs 

Elasticity 
of  Labor 

Elasticity 

of 

Duration 


Percentage  of 
Traceable 

Measures 

Days  to  Plan 
Evaluation 

X^ning 

Outputs 

Elasticity  of 
Labor 

Elasticity  of 
Duration 

Overall 

Utility 

(0-1) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.16 

0.71 

JTEM  CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 
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Quantitative  Modeling  Matrix 


Percentage 

Traceable 

Measures 

Evaluation 

Planning 

Duration 

Planning 

Output 

Quality 

Labor 

Elasticity 

Duration 

Elasticity 

Overall 

Utility 

(0-1) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.17 

0.71 

JTEM 

CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 
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Utility  &LCCE 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality 

of 

Planning 

Outputs 

Elasticity 
of  Labor 

Elasticity 

of 

Duration 

Overall 

Utility 

(0-1) 

LCCE 

($M) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

5.01 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.17 

0.71 

5.98 

JTEM 

CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

6.97 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

7.72 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 

8.13 
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Utility 


LCCE  vs  Utility 

iBf 

0.90 

0.80 

0.70 

0.60 

0.50 

0.40 

0.30 

0.20 

0.10 

0.00 

$-  $2.00  $4.00  $6.00  $8.00  $10.00  $12.00  $14.00 

Life  Cycle  Cost  Estimate  (SMIL) 
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LCCE  vs  Utility 


®JHrSL 


0.95 
0.9 
0.85 
0.8 
£•  0.75 
1  0.7 

0.65 
0.6 
0.55 
0.5 

4  5  6  7  8  9 

Cost  $M 
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Way  Ahead:  3  areas 


Strategic 

Functional  Area 

Policy 

1  Joint  Operations  Concepts 

Guidance 

| Integrated  Architectures! 
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Refined  Problem  Statement  68 


"There  is  no  system  that  defines  and 
compares  System  of  System  performance 
measures  to  war-fighter  needs  in  an 
objective  and  measurable  way." 


War  Fighter 
Needs 


Are  they  aligned? 


SoS 

Performance 

Measures 


Individual  System 
Design  Spec 


JC3M  -  Paper  5407 


39 


Federation  Of  Systems  (FEDOS) 


> 


Elicit  Requirements 
from  Service 
Stakeholders  for  each 
event: 

“AFATDS  must  display 
unit  symbology” 


Did  AFATDS  report  ammo 
status  correctly? 

Did  EPLRS  transmit  firing 
data? 


Service 

System  “Owners” 


System-Centric 

Testing 


Service  Test 
Organization 


System  Requirements 
System  Test  Plan 
System  Test  Procedures 
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Marine  Air  Ground  Task  Force  C4I  Capability 
Cejtificatior^ _ 


Service  Test 
Organization 


SoS  Capability  Assmt  Plan 
SoS  Performance  Measures 


Capabilities  Package 
from  Stakeholders  for 
each  event: 

“AFATDS  must  send  msg 
to  TBMCS...” 


Service 

Doctrine  Developers 
System  “Owners” 


SoS  Capability 
Assessment 

Was  Call  For  Fire: 
Timely 
Reliable 
Accurate... 
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Joint  Test  &  Evaluation  Methodology  Capability 


1  j-t ■  r-fc*  TaoI  _ _ K 

Pgm  Introduction  Doc: 

*  SoS,  SUT,  Environment, 
JOC,  COI,  MOP,  MOE 

•  SoS  Evaluation  Strategy 
Test  Plan 

Joint  Test  \ 

Organization  v 

Review  Joint 

Doctrine, 

CONOPS,  System 

Documentation  for 
each  event 

£ 

SoS  Capability 
Assessment 

Was  Call  For  Fire  effective  in  a 
Joint  Mission  environment? 

Is  XXX  an  appropriate 
investment? 
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Functional  Capabilities  Board  (FCB) 


Joint  Test 
Organization 


Define  SoS 

Performance  Measures 
(ongoing) 


JCIDS  C2  FCB, 
System 

Documentation 


SoS  Performance  Measures 

SoS  Test  Plan 

SoS  Test  Procedures 


SoS  Capability 
Evaluation 

Was  speed  (accuracy, 
effectiveness,  efficiency...) 
improved,  unchanged,  or 
degraded? 
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System  Capabilities  Review  (SCR) 


Joint  Test 
Organization 

k 

SoS  Performance  Measures 
SoS  Test  Plan 

SoS  Test  Procedures 

> 

V 

Review  Joint 

Doctrine, 

CONOPS,  System 

Documentation  for 
each  event 

SoS  Capability 
Evaluation 


Was  speed  (accuracy, 
effectiveness,  efficiency...) 
improved,  unchanged,  or 
degraded? 
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Blank  Scoring  Matrix 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality 

of 

Planning 

Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

1.3.2 

1.4.3 

1.4.3 

4.1 

4.1 

FEDOS 

MC3T 

JTEM  CTM 

FCB 

SCR 
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CORE 


Functional  Test  Requirments 


Test  Procedure  Comments 
HW/5W  Delivery  Dates 
CONOP5  and  Architecture  Diagrams 

Systems  List,  Project  Schedule,  HW/SW  List,  Eval  Plan,  and  Metric/Grading  Process 

FCB  Measurements 

Desired  Date  tor  Test  Results  deli  very  _ 


'X 


r^K- 


Customer  Input  -  POCs^?- 


Customer  Comments  on  Thread  Arch  Diags 
Customer  Comments  on  Test  Thread  Description 
Comments  on  Metrics  and  Grading  Process 


Customer  Reviews 


List  Of  Functional  Test  Requirments 

\ 


.1.2 


Define 

Components 


Final  Test  Procedures 


Draft  Systems  List 


High  Level  Draft  Arch 


¥TT 


Diag 


Define  Evaluation 
Criteria 


Metrics  and  Grading  Process 


Draft  Test  Plan 


tTT 


Ensure 

Evaluation 

Readiness 


-►  Final  Test  Plan 


Ti) 

Candidate  C4I  System  Planning  Team 
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POW-ER 


EliciLH&S 

1-5 

U.; 

1.1.4 

- 4  1.1.5  < 
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Arena 


Baseline  Planning  Process  Model 

To  validate  this  nxxlei  with  r«l-w«ld  r  tsn  hwn,  u$e  19  systems.  4  riew  capabilities.  ai*d  10  Ctfd  capabilities 
Output  needs  to  be  wrthn  5%  of:  &,4S2  TptafTrnc  ri  Man  Itcit? 
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Conclusions 


□  JTEM  CTM  "wins" 

■  Highest  score,  but  .  .  . 

■  .  .  .  not  by  much 

□  JTEM  CTM  cost 

■  High  development:  $3.5M  vs  $2.3M 

■  Lowest  O&S:  $121, 000/year 
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Information  Modeling  for 
Systems  Integration 


bbi2.com 


Claudia  Rose 
(619)  997-5492 

Alan  Brenner 
(858)  354-2000 
Al.brenner@bbi2.com 

BBII 

PO  Box  90182 
San  Diego  CA  92169 


Introduction 


The  Information  Model  presented  was  developed  to  provide  an  enterprise  solution  to 
information  management.  It  provides  a  map  to  the  application  and  integration  of 
program  and  system  elements.  The  model  is  tool  independent,  it  provides  a 
guide  for  modeling  and  simulation  applications,  tool  capabilities,  tailoring  and 
deployment.  Model  elements  and  results  embedded  within  the  Information 
Model  and  tools  are  customized  to  automate  the  workflow  defined  in  the  model 
including  the  production  of  work  products. 

Following  this  modeling  approach  creates  a  daily  work  environment  that  facilitates 
integrated  data  development  following  preferred  processes  and  reflecting 
modeling  results  in  further  proposals  and  products.  Once  the  workflow  and 
processes  become  an  integral  part  of  the  data  development  it  becomes  easier  to 
understand  the  impact  of  discoveries  and  changes  on  the  program.  This  in  turn 
supports  ease  of  identifying  solutions  to  integration  and  development  problems. 
Ingenuity  in  design  allowing  program  development  to  utilize  existing  structures 
in  new  ways  is  enabled  through  this  approach. 

In  addition  using  an  information  model  approach  allows  simultaneous  “live”  views 
of  the  data  from  different  concerns  including  management  and  IPTs.  The 
inclusion  of  program  concerns  such  as  Risk  and  Test  gives  a  more  complete 
response  to  problems  and  issues  in  those  areas. 
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About  Us 


BBII  provides  experts  in  Systems  Engineering  and  Architecture.  The  company  has  developed 
an  Information  Model  approach  to  integrating  program  functions.  Customers  include 
Bombardier,  Northrop  Grumman,  NASA,  SAIC,  Sikorsky,  the  State  of  Texas,  ViaSat,  and 
others.  BBII  has  maintained  partnerships  with  a  variety  of  tool  vendors.  BBII  can  provide 
a  team  to  identify  the  model,  modify  the  tools,  write  instructions,  mentor  and  train  staff, 
develop  data,  provide  systems  engineers,  systems  architects  and  engineering  support. 


Claudia  Rose  is  the  president  and  creator  of  BBII,  a  Systems  Engineering  Consulting  and 
Support  Company.  She  has  presented  papers  on  Systems  Engineering  tools  and  processes 
at  INCOSE,  NDIA  and  AFCEA  conferences  and  others.  She  has  served  on  boards  of 
directors  in  recent  years  that  include  INCOSE  San  Diego,  NDIA  small  business  forum, 
AUVSI  and  the  La  Jolla  Cove  Swim  Club.  She  holds  an  MAIT  (Master  International 
Transactions)  from  George  Mason  University,  with  studies  Tribhuvan  University 
Kathmandu,  and  a  BA  from  the  University  of  Wisconsin-Madison.  Her  research  has 
focused  on  bringing  order  out  of  chaos.  She  has  worked  as  a  consultant  on  Health  and 
Development  projects  at  The  World  Bank  and  USAID,  presented  papers  on  the  health 
development  policy  process,  created  databases  for  canning  companies  and  personal 
trainers,  before  bringing  the  special  organization  credo  of  BBII  to  the  world  of  systems 
engineering. 
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Information  Model  Benefits 


•  OPTIMAL  DESIGN 

-  The  information  model  facilitates  a  design  where  gaps  in  the  satisfaction  of 
operational  needs  drive  an  adaptive  solution  to  reduce  the  gap 

-  The  information  model  facilitates  the  development  of  alternative  approaches  at  higher 
levels,  up  to  re-characterization  of  the  operational  needs,  to  allow  an  overall  design 
solution  which  better  satisfies  the  operational  needs  of  the  platform 

-  Characterized  by  measures  of  effectivity 

•  ENTERPRISE  ARCHETECTURE 

-  Tie  together  stakeholders  and  represent  their  needs 

-  Tie  together  System  Elements 

-  Integrate  Management 

•  WORKFLOW  AUTOMATION 

-  Allows  information  to  be  viewed  in  its  entire  context 

-  Work  products  including  specifications  and  reports  are  produced  as  byproducts  of  the 
database 

-  Collaboration  is  supported  as  part  of  the  workflow 

•  DESIGN  ASSURANCE 

-  Disciplined  Systems  Engineering  process 

-  Validation 

-  Verification 
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Why  this  Approach? 


•  Providing  the  best  value  solutions 

•  Use  of  modeling  and  tools  that  allow  team  members  to  collaboratively 
integrate  their  work  with  the  entire  program 

•  Collaborating  to  produce  better  options  with  existing  resources 

•  Finding  new  ways  to  accomplish  new  objectives  within  existing 
framework 

•  Identifying  and  evaluating  options  throughout  the  program  development 
process 

•  Re-characterize  statements  of  need  and  higher  level  requirements  to 
allow  innovative  and  ingenious  solutions 
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Information  Model  (Sample) 


Primary 

Source 

Document 


[5  5  & 

Standards,  Specifications  &  Policies 


DoD-AF 


www.hkji2.com 


Information  Model  (Sources) 


Primary 

Source 

Document 


C) 


Via  interviev 


Pi  r 
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Standards,  Specifications  &  Policies 


A  \ 


v 


nJnc^ri 


System 

View 


Design 
►  Software 
Architecture 

(See  Additional 
Chart) 


E 


□  □ 


Symbolically  Referenced 
External  Drawings  &  Files 


Component  Architecture 


SETS: 


Work  Breakdown  Structure  View 
Work/Cost/Management 
Taxonomy 


Interface  Hierarchy  & 
ICD  Document  Structure 


a  i' 

CD  Document  Tree 

Sorip, 

elf 


Management  View  and  Standards 


TV-2  Standards 
Technology 
Forecast 

Description  of 
emerging  standards 
that  are  expected  to 
apply  to  the  given 
architecture,  within 
an  appropriate  set 
of  timeframes 


Work  Breakdown  Structure 
Work/Cost/Management 
Taxonomy 
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MODELS 


Functional  Flow  Block 
Diagrams 


Design 

Software 

Architecture 

(See  Additional  Chart) 


Trade  Studies 


Function  Performed 


Functional 


Decomposition 


System 


System  View 
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Li 
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Interface  Flierarchy  & 
ICD  Document  Structure 


1A1 

ICD  Document 
Tree 


[ 

1 

0 

0 

r 

C 

use  uases 
Activity  Diagrams 
Sequence  Diagrams 
State  Diagrams 
Performance  Models 
Executable  Models 


Symbollic 

references 


Generates 


Component  Architecture 


Operational  View 
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Decomposition 
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Behavioral  Decomposition 


TA 

OF 


Decomposition  OV  examples 
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0V-5  Operational  Activity 
Model 

Activities,  relationships 
among  activities,  I/Os, 
constraints  (e.g.,  Policy, 
guidance),  and  mechanisms 
that  perform  those  activities. 
In  addition  to  showing 
mechanisms,  overlays  can 
show  other  pertinent 
information. 


OV-7  Logical  Data 
Model 

Documentation  of  the 
data  requirements 
and  structural 
business  process 
rules  of  the 
Operational  View 


OV-1  High-level 
Operational 
Concept  Graphic 

High-level  graphical 
description  of 
operational  concept 
(high-level 
organizations, 
missions,  geographic 
configuration, 
connectivity,  etc.) 


OV-2  Operational  Node  Connectivity 
Description 

Operational  nodes,  activities  performed 
at  each  node,  connectivity  and 
information  flow  between  nodes 


OV-3  Operational  Information  Exchange 
Matrix 

Information  exchanged  between  nodes  and 
the  relevant  attributes  of  that  exchange 
such  as  media,  quality,  quantity,  and  the 
level  of  interoperability  required 
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Component  Architecture 


SV-1  System  Interface 
Description 

Identification  of  systems 
and  system  components 
and  their  interfaces,  within 
and  between  nodes. 


SV-7  System  Performance 
Parameter  Matrix 

Performance  characteristics  of 
each  system (s)  hardware  and 
software  elements,  for  the 
appropriate  timeframe(s) 


_  £& 

Tables  Diagram 

Interface  Hierarchy  & 
ICD  Document  Structure 


ICD  Document  Tree 


Report 

Script 


Test 


Behavioral 
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Risk 


Risk  Mitigation  Planning 
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Risk 


SV-8  System  Evolution  Description 

Planned  incremental  steps  toward 
migrating  a  suite  of  systems  to  amore 
efficient  suite,  or  toward  evolving  a 
current  system  to  a  future  implementation 


Risk  Mitigation  Planning 


SV-9  System  Technology  Forecast 

Emerging  Technologies  and  software/ 
hardware  products  that  are  expected  to 
be  available  in  a  given  set  of 
timeframes,  and  that 


Proposal 


Keys  to  Successful  Information 
Model  Implementation . 

•  Reuse  of  (tailored)  tools  and  models  for  each  deployment 

•  Understanding  the  impact  of  changes  on  the  entire  program 

•  Processes  which  facilitate  innovative  changes 

•  Integrated  work  environment 

•  Ability  to  translate  change  at  one  level  to  changes  at  all  levels  within  the  WBS 

•  Understanding  what  needs  to  happen  (Operational  Requirements) 

•  Identification  of  gaps  between  operational  needs  and  selected  approach 

•  Value  system  to  support  focus  on  narrowing  gaps  with  most  impact 

•  Infrastructure  which  encourages  and  supports  alternative  approaches  which  can 
better  satisfy  higher  level  needs 

•  Infrastructure  which  supports  rapid  evaluation  of  the  value  and  impact  of 
alternative  approaches 

•  Thinking  outside  the  scope  of  current  solutions 
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Support  Structure 

•  Integrated  Information  Model  to  facilitate  common  understanding  and 
collaborative  work  environment 

•  Operational  Requirements 

-  Operational  models 

-  Flexibility  to  restate  operational  models  and  capabilities  to  meet  original 
objectives  with  alternative  approaches 

-  Ability  to  recognize  the  value  of  enhanced  or  new  capabilities 

•  Linkage  of  operational  needs  to  design  requirements 

•  Operational  models 

-  Facilitate  understanding  of  needs 

-  Organize  information 

•  System  and  design  models 

-  Facilitate  understanding  of  system  and  design 

-  Organize  information 

•  Continuous  validation 

•  Measures  of  Effectivity  and  a  Value  System 

•  Best  Practices 

•  Lessons  Learned 
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Validate  Requirements  to  Satisfy 

Operational  Needs 


•  Validation  is  a  continuous  ongoing  process  to  make  sure  the  right  thing  is 
being  done 

•  Capturing  Satisfaction  Arguments  as  the  analysis,  decomposition  and 
design  proceeds  identifies  gaps  early  at  a  time  they  can  be  more  easily 
resolved 

•  Measures  of  effectivity  can  be  integrated  with  satisfaction  arguments 

•  Formal  Validation  will  tie  together  elements  of  the  Information  Model  to 
validate  that  the  operational  needs  are  satisfied. 
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Conclusion 


An  Information  Model  based  approach  supports  an  optimal  design 
enhancing  program  capabilities.  It  drives  a  collaborative  work 
environment  reducing  rework,  revealing  issues  and  supporting  needed 
changes  in  an  efficient  manner. 

The  Information  Model  approach  provides  a  roadmap  for  enterprise 

development  through  integration  of  corporate  knowledge  and  experience. 
It  supports  the  information  maturity  processes  through  their  integration  in 
elements  of  daily  workflow.  It  reduces  rework  in  preparation  of  work 
products  and  in  the  work  process. 

Models  are  key  to  both  characterizing  System  Performance  and  relating  this 
to  the  operational  needs  through  the  measures  of  effectivity 
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November  12-15,  2007 


A  Methodology  for  Assessing  and  Prioritizing 
The  Risks  Associated  with  The  Level  of 
Verification,  Validation  and  Accreditation  Of 

Models  And  Simulations 


Track  5:  Virtual  Testing  /  Modeling  and  Simulation  in  the  Collaborative 

Environment 


James  N.  Elele,  Ph.D. 

NAVAIR  -  Integrated  Battlespace  Simulation  and  Test  (IBST)  Department 

481 50  Shaw  Road 
Building  2109 
Patuxent  River,  MD  20670 
301-342-4154 
James.Elele@Navv.mil 


•  Modeling  &  Simulation  (M&S)  are  integral  to  the  Defense  Acquisition  process  in 
the  United  States 

•  For  M&S  to  be  useful  tools  in  acquisition,  they  must  be  credible  and  suitable  to 
the  specific  intended  use(s)  of  interest 

•  Verification,  Validation  and  Accreditation  (W&A)  helps  to  reduce  risk 
associated  with  M&S  use  by  establishing: 

-  Whether  a  particular  M&S  and  its  input  data  are  credible  and  suitable  for  a 
particular  task 

-  Based  on  objective  evidence 

•  DoD,  Service  and  Operational  Test  Agency  (OTA)  policy  require  W&A  for  M&S 
used  to  support  acquisition 

-  DoDI  5000.61 ,  SECNAVINST  5200.40,  COTFI  5000.1  A 

•  Resources  are  limited,  so  you  need  a  logical  way  to  guide  your  investment  in 
model  credibility  and  W&A 

-  How  much  effort  to  expend  establishing  credibility  and  suitability  of  your  M&S 
toolbox  (supporting  VV&A) 

-  How  best  to  invest  resources  to  get  the  most  return  on  investment  and  add  the 
most  value 
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M&S  and  Risk  in  Policy 


•  All  W&A  implementing  policies  we’re  aware  of  indicate  that  the  magnitude 
of  the  effort  to  support  accreditation  should  be  commensurate  with  risk 

-  DoDI  5000.61,  SECNAVINST  5200.40  ... 

•  But  —  little  practical  guidance  is  given  in  these  high  level  policies  on  how 
to  actually  do  this 

•  This  briefing  describes  a  general  approach  developed  by  the  Joint 
Accreditation  Support  Activity  (JASA)  to  establishing  a  cost  effective 

risk-based  W&A  strategy  for  acquisition  programs: 

-  Consistent  with  policy 

-  Based  on  experience  with  successful  M&S  accreditation  efforts 

-  Consistent  with  the  Defense  Modeling  and  Simulation  Office’s  VV&A 
Recommended  Practices  Guide  (RPG) 

-  Incorporating  industry  standards  and  best  practice 


N  AV 


Our  Approach 


•  Flexible  and  Proven  Approach  : 

-  Is  consistent  with  VV&A  policy  and  the  Defense  Modeling  and 
Simulation  Office’s  VV&A  Recommended  Practices  Guide 
(RPG) 

-  Is  based  on  experience  with  successful  M&S  accreditation 
efforts,  supporting  major  acquisition  programs  (e.g.  PMA-261 
CH-53K,  VH-71,  &  P-8A  Multi-mission  Aircraft  program) 

-  Reflects  industry  standards  and  best  practice 

-  Incorporates  risk-based  accreditation  methodology  developed 
by  Joint  Accreditation  Support  Activity  (JASA)  over  more  than 
a  decade 

-  Builds  on  structures  and  practices  already  in  place  in  DoD 
acquisition  program  (program’s  existing  risk  management 
approach,  working  group/IPT  structure,  delegation 
agreements,  etc.) 


What  is  Risk? 


In  the  risk  management 
community,  risk  is  generally 
defined  as  the  likelihood  that 
something  (usually  bad)  will 
happen  times  the 
consequences  if  it  does 

-  Sometimes  in  casual 
speech  people  use  the 
word  “risk”  to  mean 
likelihood  of  occurrence 


To  reduce  risk,  either  reduce 
the  likelihood  that  something 
will  occur  or  reduce  the 
severity  of  the  consequence 

-  Risk  literature  also 
discusses  the  idea  of 
exposure,  which  we’ll 
come  back  to  shortly 


RISK  =  LIKELIHOOD  X  CONSEQUENCE 

To  reduce  risk,  reduce  either 
likelihood  or  consequence 


RISK 


Risk  Associated  with  M&S  Use 


Here,  the  risk  of  interest  is  the  risk 
associated  with  using  M&S 

-  M&S  includes  the  models  and 
simulations  as  well  as  the 
necessary  input  data 

Likelihood  is  the  odds  that  the  M&S 
and/or  their  input  data  are  incorrect 
or  inappropriate  to  your  intended 
use 

Consequence  is  the  impact  if  the 
M&S  output  is  wrong  but  you 
believe  it  and  act  on  it 


Note:  The  risk  associated  with  model 
development  -  will  it  be  done  on  time  and 
within  budget — is  an  important  but  separate 
issue.  Here  we  focus  on  operational  risk. 


RISK  =  LIKELIHOOD  X 
CONSEQUENCE 


HIGH 


MEDIUM 


LOW 


Likelihood 
M&S  are 
wrong 


Consequences  if 
M&S  are  wrong 


] 


Consequence  of  a  Poor  Decision 
vs.  Consequence  if  Model  is  Wrong 


•Consequence  if  model  is  wrong  depends 
on: 

•Role  M&S  play  in  the  decision-making 
process 

•Consequence  of  a  poor  decision 


RISK 


HIGH 


Likelihood 
that  M&S  is 
wrong 


Consequence 
if  M&S  is 
wrong 


• Here ,  the  role  of  M&S  in  decision 
making  is  similar  to  the  concept  of 
exposure  in  the  risk  literature 

•Reduce  risk  by  limiting  exposure 

•One  way  to  reduce  the  risk 
associated  with  M&S  use  is  by 
limiting  the  role  of  M&S  in  the 
decision  process 


Consequence  if  model  is  wrong  = 
f  (role  of  M&S  in  decision 
and 

consequence  of  poor  decision) 


So  Here’s  the  Point ... 


•  Risk  associated  with  use  of  M&S  is  driven  by  likelihood  M&S  is  wrong  and 
consequence  thereof 

•  VV&A  addresses  likelihood  of  M&S  error  (and  thus  confidence  in  model  results) 

-  Level  of  risk  you  can  accept  and  consequences  if  model  is  wrong  drive  the 
amount  of  effort  required  to  establish  an  acceptable  level  of  confidence 

-  Also,  likelihood  M&S  is  wrong  and  consequence  if  the  model  is  wrong  drive 
risk  you  accept  if  you  use  M&S 


•  If  you  had  a  practical  method  of  apply  these  principles ,  you  could  determine  how 
much  effort  to  put  into  VV&A 

-  What  kind  and  how  much  evidence  is  required  to  establish  confidence  and 
reach  accreditation  decision  for  particular  uses 

-  Extent  of  appropriate  review  process  Drive 

-  Level  of  independence  in  V&V  and  review  Resources 

-  Appropriate  level  of  accreditation  authority 

•  This  briefing  offers  you  one  approach  to  consider  and  some  implementation 
suggestions 


Considerations/Practical  Problems 


•  Problem:  You  can’t  always  (or  even  often)  come  up  with  actual  numbers  for  either 
consequence  (cost,  lives  lost,  etc.)  or  likelihood,  so  how  can  you  multiply  what  you 
don’t  have? 


-  Solution: 

•  Usually  resort  to  using  estimates  within  defined  bands  or  levels  or  bins: 
High,  Medium,  Low,  etc. 

•  Adopt  a  scheme  for  combining  levels  to  arrive  at  a  single  value  (combine 
likelihood  value  and  consequence  value  to  get  risk  value) 

•  System  Safety  community  has  some  practical  ideas  we’ll  show  you 
•  Heads  up: 

-  Current  DoD  and  Navy  VV&A  policy  discusses  certain  circumstances  in  which 
formal  accreditation  of  M&S  is  required  (DoD  5000.61,  SECNAVINST  5200.40) 

-  Updated  Navy  policy  will  require  ALL  M&S  in  use  in  the  Navy  as  of  the  effective 
date  of  the  instruction  to  be  verified,  validated  and  accredited  (proposed 
SECNAVINST  5200.40A) 

-  Your  strategy  needs  to  have  provisions  in  case  5200.40A  comes  into  effect 
during  the  life  of  your  program 


Tools  of  the  Trade 


•  You’ll  need  scales  and  rules 

-  Scale  and  selection  criteria  for 

•  Levels  of  risk  associated  with  M&S  use 

•  Levels  of  likelihood  of  error  (and  an  inverse  scale  for  the  level  of 
confidence  in  M&S  results) 

•  Levels  of  consequence  if  model  is  wrong 

•  Levels  for  role  of  M&S  in  decision  making 

•  Levels  of  consequence  if  decision  is  poor 

-  Level  combining  rules 

•  Combine  (role  of  M&S  in  decision  making)  &  (level  of 
consequence  of  a  poor  decision)  to  get  (Level  of  consequence  if 
model  is  wrong) 

•  Combine  (likelihood  of  model  error)  &  (level  of  consequence  if 
model  is  wrong)  to  get  (risk  level) 


•  And  you’ll  need  Tables 

-  Nature  and  extent  of  information  necessary  to  support  accreditation 
as  a  function  of  acceptable  likelihood  of  M&S  error  (or  required  level 
of  confidence) 

-  Method  of  developing  accreditation  recommendation  given  level  of 
consequence  of  M&S  error 

-  Approval/signature  authority  given  level  of  consequence  of  M&S  error 

•  The  next  few  slides  give  a  quick  trip  through  the  method  (scope  W&A 
effort)  and  (estimate  risk  given  a  decision  to  use  a  model  as  is)  to  give  you 
a  feel  for  how  the  tools  are  used 

•  Then  we’ll  look  at  notional  samples  of  each  tool 

•  Then  we’ll  discuss  some  examples  of  how  these  ideas  have  been  used  in 
successful  accreditation  efforts 


Goal  #1 :  How  much  VV&A  is  necessary 
to  support  accreditation? 


Key:  If  you  know  this,  you  can 


1.  Define  intended  use  (decision  supported  by  M&S) 

2.  Determine  role  of  M&S  in  the  decision  process  and  pick  appropriate  value  from  role  table 

3.  Assess  consequence  if  the  decision  is  poor  and  pick  the  appropriate  value  from  decision 
consequence  table  (Consequence  of  decision) 

4.  Determine  what  level  of  risk  the  decision  maker  is  willing  to  assume  for  this  particular  use 

of  M&S  (Acceptable  Risk) 

5.  Use  role/decision  consequence  table  to  determine  a  value  for  consequence  if  the  model  is 
incorrect  (Consequence  if  M&S  wrong) 

6.  Use  Likelihood  of  error/decision  consequence  table  to  determine  the  highest  likelihood  of 
error  value  that  will  result  in  the  acceptable  level  of  risk  given  the  consequence/M&S  wrong 


7.  Look  at  the  VV&A  evidence  table  to  determine  what  kind  and  how  much  information  is 
necessary  to  support  an  accreditation  assessment,  given  the  likelihood  of  error  value  from 
step  6. 


•  8.  Look  at  the  Accreditation  Recommendation  table  to  determine  what  approach  will 
be  taken  to  generate  an  accreditation  recommendation,  given  the  consequence-M&S 
wrong 

•  9.  Look  at  the  Decision  Authority  table  to  determine  the  signature  authority  for  VV&A 
plans  and  reports  as  well  as  the  accreditation  decision  authority. 

•  10.  Use  answers  in  7,  8,  and  9  to  develop  a  workable  plan  to  gather/generate  required 
information  package,  generate  an  accreditation  recommendation,  and  come  to  an 
accreditation  decision 


Goal  #2:  How  much  risk  is  associated  with 
M&S  use,  given  the  evidence  available? 


Key:  If  you  know  this,  you  can  figure  this  out 


Reality  Bites:  You  have  no  choice  of  M&S  and  you  have  no  time  or  resources  for  additional 
V&V.  Here’s  how  to  get  a  handle  on  the  risk  associated  with  model  use. 


•  You’ll  need  to  do  some  research  first 

•  1.  Gather  the  VV&A  related  information  that  is  available,  look  at  the  likelihood  of  model 
error  table,  and  determine  roughly  which  level  the  nature  and  amount  of  information  you 
have  equates  to — this  gives  you  the  likelihood  of  error  value 

•  Then  you’ll  need  to  know  some  key  characteristics  about  the  situation  under 
consideration 

•  2.  Define  intended  use  (decision  supported  by  M&S) 

•  3.  Determine  role  of  M&S  in  the  decision  process  and  pick  appropriate  value  from  role 
table 

•  4.  Assess  consequence  if  the  decision  is  poor  and  pick  the  appropriate  value  from 
decision  consequence  table  (consequence  of  poor  decision) 


Goal  2  (continued) 


Then  determine  the  level  of  consequence  if  the  model  is  wrong 

•  5.  Use  the  role  of  M&S  level  from  Step  3  and  the  consequence  of  poor  decision  level  from 
Step  4  to  determine  the  level  of  consequence  if  the  M&S  is  wrong  from  the 
role/consequence  of  model  error  table. 

Then  you  can  back  out  level  of  assumed  risk 

•  6.  Use  likelihood  of  error/consequence  of  decision  table  to  back  out  the  level  of  risk 


•  Clearly  not  the  ideal  situation,  but  it  happens  quite  frequently. 

•  Even  if  you’re  stuck  using  the  (less  than  ideal)  tool  you  have,  the  boss  needs  to 
have  a  feel  for  how  much  confidence  to  place  in  the  answers 

•  Path  2  gives  you  a  way  to  estimate  risk 


Scales,  Rules  and  Tables 

-  Examples 

-Some  Tips  and  Advice 


Here’s  an  example  of  a  risk  scale  with  three  levels 

-  Many  programs  use  a  three  level  high/medium/low  risk  scale 

-  Very  conducive  to  the  use  of  stoplight  charts 

Give  strong  consideration  to  starting  with  the  risk  level  structure 
already  in  use  on  your  program  and  adapting  it  for  use  in  your  W&A 
approach 


Risk  Level 

- w 

Definition 

High 

Unacceptable.  Major  disruption  likely.  Different  approach  required.  Priority 
management  attention  required 

Moderate 

Some  disruption  may  occur.  Different  approach  may  be  required.  Additional 
management  attention  may  be  needed 

Low 

r  i 

Minimum  impact.  Minimum  oversight  needed  to  ensure  risk  remains  low. 

: _ >! 

Levels  of  Confidence  /  Likelihood  of 

M&S  Error 


•  Here’s  one  suggestion  based  upon  JASA’s  experience  and  guidelines  in 
DMSO  VV&A  RPG 


Include  one  level  for  either  low  or  unknown  level  of  confidence  so  that  your  approach 
has  a  minimal  effort  option  to  cover  emergency  or  low  consequence  situations 


Likelihood 
of  Error 

Confidence 

Level 

Description 

1 

4 

Very  high  confidence  based  upon  extensive 
documented  V&V  relevant  to  intended  use 

2 

3 

High  confidence  based  on  face  validation  by  SMEs 

3 

2 

Moderate  confidence  based  upon  previous  usage 
history 

4  (High) 

1 

Low  or  unknown  level  of  confidence.  M&S  appears  to 
have  the  functionality  required  but  credibility  is 
unknown. 

Levels  of  Consequence 


•  Here’s  an  extremely  simple  example  of  consequence  levels  with  four  broadly 
defined  levels 


\  Whatever  scheme  you  choose,  you  should  make  provisions  to  consider 

consequences  of  varying  natures  including  cost,  schedule,  personnel  safety, 
yQ  political,  operational 

-  Also  be  sure  you  take  into  consideration  all  of  the  ways  the  model  output 
could  be  wrong  (e.g.  M&S  could  erroneous  over-  or  under-estimate 
performance  of  a  military  system,  and  the  consequences  might  be 
different  for  each  case) 


Consequence 

Level 

- w 

Definition 

High 

Major  disruption  to  program.  Different  approach  required.  Priority 
management  attention  and  resource  allocation  required  immediately. 

Moderately 

High 

Significant  disruption  to  program.  Different  approach  required.  Priority 
management  attention  required. 

Moderate 

Noticeable  disruption  Different  approach  may  be  required.  Additional 
management  attention  may  be  needed. 

Low 

f 

r  Minimum  impact.  Minimum  oversight  needed  to  ensure  risk  remains  low. 

Levels  of  Consequences  if  Decision  is  Poor 


Level 

Technical  Performance 

Schedule 

Cost 

5 

Severe  degradation  in  technical 
performance;  cannot  meet  KPP  or  key 
technical/supportability  threshold;  will 
jeapardize  program  success;  no 
workarounds 

Cannot  meet  key 
program  milestones 

Slip> _ months 

Exceed  APBA  threshold 

>  (1 0%  of  budget) 

4 

Significant  degradation  in  technical 
performance  or  major  shortfall  in 
supportability;  may  jeapardize  program 
success;  workarounds  may  not  be 
available  or  may  have  negative 
consequences 

Program  critical  path 
affected,  all  schedule 
float  associated  with  key 
milestone  exhausted 

Slip< _ months 

Budget  increase  or  unit 
production  cost 
increases 

<(10%  of  budget) 

3 

Moderate  reduction  in  technical 
performance  or  supportability  with  limited 
impact  on  program  objectives; 
workarounds  available 

Minor  schedule  slip,  no 
impact  to  key  milestones 

Slip<month(s)  of  critical 
path 

Sub-system  slip> _ 

months(s) 

* 

Budget  increase  or  unit 
production  cost 
increases 

<  (5%  of  budget) 

2 

Minor  reduction  in  technical  performance 
or  supportability,  can  be  tolerated 
withlittle  or  no  impact  on  program;  same 
approach  retained 

Additional  activities 
required,  able  to  meet 
key  dates 

Slip< _ months  (s) 

Budget  increase  or  unit 
production  costs 
increases 

<(1%  of  budget) 

„  1  , 

r  Minimal  or  no  impact  ^ 

r  Minimal  or  no  impact  ^ 

Minimal  or  no  impact 

1 - - - - - K 

V A  I 


Here’s  a  Complicated  Scheme  for  “Quantifying” 
Consequence  (Impact)  of  Poor  Decision 


From  MIL-STD  882C/D  on  System  Safety 


Impact 

Categories 

Impact  Level: 
Catastrophic 

Impact  Level: 
Critical 

Impact  Level: 
Marginal 

- w 

Impact  Level: 
Negligible 

Personnel  Safety 

Death 

Severe  Injury 

Minor  Injury 

w 

<  Minor  Injury 

Equipment  Safety 

Major  Equip  Loss’ 
Broad  Scale  Major 
Damage 

Small  Scale  Major 
Damage 

Broad  Scale  Minor 
Damage 

Small  Scale  Minor 
Damage 

Environmental 

Damage 

Severe  (Chernobyl) 

Major  (Love  Canal) 

Minor 

Some  Trivial 

Occupational 

Illness 

Severe  &  Broad 

Severe  or  Broad 

Minor  and  Small 
Scale 

Minor  or  Small 
Scale 

Cost 

Loss  or  Program 
Funds;  100%  Cost 
Growth 

Funds  Reduction; 

50%  to  1 00%  Cost 
Growth 

20%  to  50%  Cost 
Growth 

<20%  Cost 

Growth 

Schedule 

Slip  Reduces  DoD 
Capabilities 

Slip  Causes  Cost 
Impact 

Slip  Causes 

Internal  Turmoil 

w 

Republish 

Schedules 

Political 

Nat’l  or  Internat’l 
(Watergate) 

Significant  (Tailhook) 

Embarrassment 
($200  Hammer) 

w 

Local 

Operational 

r  i 

Widespread  Add’l 
Combat  Deaths 

r  i 

Limited  Add’l  Combat 
Deaths 

f  i 

Moderate  Add’l 
Casualties 

r  i 

w 

Minimal  Add’l 
Casualties 

f  J 

JO 


Role  of  M&S  in  Decision  Making 


•  Here’s  an  example  scheme 


Role 

Level 

Definition 

4 

M&S  will  be  the  onlv  method  employed  to  make  a  decision 

3 

w 

M&S  will  be  the  primary  method,  employed  with  other  non-M&S  methods 

2 

M&S  will  be  a  secondary  method,  employed  with  other  non-M&S  methods, 
and  will  provide  sianificant  data  unavailable  throuah  other  means 

1 

M&S  will  be  a  supplemental  method,  employed  with  other  non-M&S 
methods,  and  will  provide  supplemental  data  already  available  throuah 
other  means 

- - w 

Combination  Schemes 
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Level 

What  is  the 

Likelihood  the  Risk 
Event  will  Happen? 

E  (High) 

Near  Certainty 

D 

Highly  Likely 

C 

Likely 

B 

Unlikely 

A 

r  i 

Remote 
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Consequence 

t 


Assigned  Risk  Level 

R 

- ► 

High  -  Unacceptable.  Major 
disruption  likely.  Different 
approach  reqd.  Priority  mgmt 
attention  reqd. 

Y 

Moderate  -  Some  disruption. 
Different  approach  may  be  reqd. 
Addl  mgmt  attention  may  be 
needed 

G 

r  i 

Low  -  Minimum  impact. 

Minimum  oversight  needed  to 

ensure  risk  remains  low. 

1 - — - 

- 

Level 

Technical  Performance 

And/ 

or 

Schedule 

And/ 

or 

Cost 

And  / 

or 

- w 

Impact  on 
Other  Teams 

A 

Minimal  or  no  impact 

Minimal  or  no  impact 

Minimal  or  no  impact 

None 

B 

Acceptable,  some 
reduction  in  margin 

Additional  resources  reqd;  able 
to  meet  need  dates 

<5% 

Some  impact 

C 

Acceptable;  significant 
reduction  in  margin 

Minor  slip  in  key  milestones; 
not  able  to  meet  need  date 

5  -  7% 

Moderate 

impact 

D 

Acceptable;  no 
remaining  margin 

Major  slip  in  key  milestones  or 
critical  path  impacted 

7-10% 

Major  impact 

E 

,  (High)  , 

Unacceptable 

r  ' 

r  ' 

Can’t  achieve  key  team  or 
r  major  program  milestones  , 

r  i 

>10% 

r  ' 

r  ' 

Unacceptable 

: - ► 

Program  Risk  Reporting 


7T 

CD 


O 

o 

Q. 


Level 

Likelihood  the  Event 

Will  Happen? 

Probability  of 
Occurrence 

5  (High) 

Near  Certainty 

-90% 

4 

Highly  Likely 

-70% 

3 

Likely 

-50% 

2 

Low  Likelihood 

-30%  ! 

hi 

r  1  i 

r  Not  Likely  1 

f  ~10%  ^ 

NAV^A  I  R 


5 

L 

M  1 

4 

L 

M 

m1 

3 

L 

L 

M 

m] 

M 

2 

L 

L 

L 

M 

1 

L 

L 

L 

L 

M 

r  i 

1 

r  i 

2 

r  i 

3 

r  i 

4 

r  ^ 

5 

: *: 
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Level  of  Risk: 


High,  Med,  or 
Low 


Level 

Technical  Performance 

Schedule 

- w 

Cost 

5  (High) 

Severe  degradation  in  technical  performance;  cannot  meet 
KPP  or  key  technical/supportability  threshold;  will 
jeapardize  program  success;  no  workarounds  available 

Cannot  meet  key  program  milestones 
Slip> _ months 

w 

Exceed  APBA  threshold 

>  (10%  of  budget) 

4 

O 

o 

3 

</> 

(D 

Significant  degradation  in  technical  performance  or  major 
shortfall  in  supportability;  may  jeapardize  program 
success;  workarounds  may  not  be  available  or  may  have 
negative  consequences 

Program  critical  path  affected,  all 
schedule  float  associated  with  key 
milestone  exhausted 

Slip< _ months 

W 

Budget  increase  or  unit 
production  cost  increases 

<(10%  of  budget) 

_ 

C  o 

CD  3 

3 

O 

0 

Moderate  reduction  in  technical  performance  or 
supportability  with  limited  impact  on  program  objectives; 
workarounds  available 

Minor  schedule  slip,  no  impact  to  key 
milestones 

Slip<month(s)  of  critical  path 
Sub-system  slip> _ months(s) 

w 

Budget  increase  or  unit 
production  cost  increases 

<  (5%  of  budget) 

2 

Minor  reduction  in  technical  performance  or  supportability, 
can  be  tolerated  withlittle  or  no  impact  on  program;  same 
approach  retained 

Additional  activities  required,  able  to 
meet  key  dates 

Slip< months  (s) 

W 

Budget  increase  or  unit 
production  costs  increases 
<(1%  of  budget) 

1 

r  ' 

Minimal  or  no  consequence  to  technical  performance 

r  1 

Minimal  or  no  impact 

r  ’ 

Minimal  or  no  impact 

l _ J 

Sample  Method  of  Generating 
Consequence  /  Evidence 
Required  to  Support  Accreditation 


Method  of  Generating  Accreditation 
Recommendation/Consequence  if  M&S  is  Wrong 


This  table  identifies,  for  each  level  of  consequence  if  the  M&S  is  wrong,  the  method  that  will  be  used  to 
come  to  an  accreditation  recommendation 

Generally,  higher  levels  of  consequence  merit  review  and  concurrence  by  major  stakeholders  (Program 
Office,  DOT&E,  OTA,  contractor)  with  support  from  appropriate  technical  SMEs 

-  The  higher  the  consequence,  generally  the  more  appearance  of  some  independent  review  becomes 
important 

.si  -  Give  strong  consideration  for  a  level  requiring  only  the  judgment  of  a  qualified  analyst  or  engineer 
^  with  minimal  (but  some)  documentation  requirements 


Consequence  Level 

- w 

Method  of  Generating  Accreditation  Recommendation 

4  (highest) 

w 

Formal  Review  of  Accreditation  Case  by  specially  convened  Accreditation 
Review  Board  resulting  in  recommendation  documented  in  formal 

accreditation  package 

3 

w 

Review  of  accreditation  case  by  M&S  IPT  resulting  in  recommendation 
documented  in  detailed  briefing  or  report 

2 

W 

Review  of  accreditation  case  by  recognized  SME  resulting  in 
recommendation  documented  in  briefing  or  report  format 

1 

r 

w 

Review  of  accreditation  case  by  responsible  engineer  documented  in 

Memo  for  the  Record 

_ j 

Example  Scheme  for 
“Quantifying”  Likelihood 


An  Example  Scheme  for  “Quantifying”  Likelihood  **The  number  of  items  should  be  specified 


Likelihood  Description 

Likelihood  of  Occurrence 
over  Lifetime  of  an  Item 

- w 

Likelihood  of  Occurrence 
Per  Number  of  Items** 

Frequent 

Likely  to  Occur  Frequently 

w 

Widely  Experienced 

Probable 

Will  Occur  Several  Times  in 
Life  of  Item 

Will  Occur  Frequently 

Occasional 

Likely  to  Occur  Some  Time  in 
Life  of  Item 

Will  Occur  Several  Times 

Remote 

Unlikely  but  Possible  to  Occur 
in  Life  of  Item 

Unlikely  but  can  Reasonably 
be  Expected  to  Occur 

Improbable 

r  ' 

So  Unlikely,  it  can  be 

Assumed  Occurrence  May  Not 
r  Be  Experienced 

Unlikely  to  Occur  but  Possible 

! t! 

Evidence  Required  to  Support 
Accreditation/Likelihood  of  Error 


For  each  level  of  likelihood  of  error  and  confidence  level,  the  table  summarizes  the  information  necessary 
to  support  an  accreditation  assessment 

-  More  rigorous  verification,  validation,  configuration  management,  discipline  in  model  development, 
and  oversight  and  review  are  required  to  drive  down  likelihood  of  error 

-  As  likelihood  of  error  goes  down,  confidence  in  model  results  goes  up 


Likelihood  of 
Error 

Confidence 

Level 

Evidence  Required  to  Support  Accreditation  Assessment 

1 

4 

Level  3  +  extensive  body  of  documented  verification  and  validation  + 
evidence  of  disciplined  M&S  development  including  history  of  technical 
and  managerial  review  over  time 

2 

3 

Level  2  +  SME  face  validation  relevant  to  current  intended  use  + 
evidence  of  effective  configuration  management 

3 

2 

Level  1  +  usage  history  +  known  V&V  history 

4  (High) 

1 

Comparison  of  M&S  requirement  derived  from  intended  use  with 
capabilities  and  limitations  of  candidate  simulation 

This  is  based  on  JASA’s  rules  of  thumb  adopted  by  the  DMSO  W&A  RPG.  See  “Role  of  Accreditation  Agent  in  W&A  of  Legacy 
Simulations”  for  more  details,  www.vva.dmso.mil 


Decision  Authority/ 
Consequence  if  the  Model  is  Wrong 


•  This  table  identifies,  for  each  consequence  (M&S  wrong)  level,  the  signature  authority  for  W&A  plans 
and  reports  as  well  as  the  accreditation  decision  authority 

•  Generally,  delegating  the  signature  and  decision  authority  as  low  as  seems  reasonable  is  the  most 
efficient  use  of  resources 

""Nt  -  DoD  and  Service  policy  give  OTAs  accreditation  authority  for  use  of  M&S  in  OT&E;  PM  for  SUT 
\  must  submit  accreditation  package  and  make  recommendation 

^  (  -  Current  practice  is  for  PM  to  be  AA  for  uses  of  M&S  within  the  purview  of  the  program  office  (e.g. 

DT&E  including  demonstration  of  spec  compliance,  LFT&E) 


Consequence 

Level 

Signature  Authority 

VV&A  Plans  &  Rpts 

- w 

Decision  Authority 

M&S  Accreditation 

4  (highest) 

Acquisition  Program  Manager  (For 
use  of  M&S  in  OT&E,  PM  is  signature 
authority  with  OTA’s  concurrence) 

w 

Acquisition  Program  Manager  (For  use 

of  M&S  in  OT&E,  OTA  is  decision  authority 
with  recommendation  from  PM) 

3 

Chief  Engineer 

Chief  Engineer 

2 

Chair,  M&S  IPT 

Chair,  M&S  IPT 

1 

r 

Responsible  Engineer  or  Analyst 

1 

w 

Responsible  Engineer  or  Analyst 

: _ 

Criticality  Analysis:  Importance  of  Decisions 


•  Descriptions  of  Level  of  Importance  of  Decision 


Level 

Description 

4 

Intended  use  addresses  multiple  areas  of  sianificant  proqram  risk,  kev  proqram  reviews 
and  test  events,  key  system  performance  analysis,  primary  test  objectives  and  test  article 
design,  system  requirements  definition,  and/or  high  software  criticality,  used  to  make  a 
technical  or  managerial  decision 

3 

Intended  use  addresses  an  area  of  siqnificant  proqram  risk  ... 

2 

Intended  use  addresses  medium  or  low  proqram  risk,  other  proqram  reviews  and  test 
events,  secondary  test  objectives  and  test  article  design,  other  system  requirements  and 
system  performance  analysis,  and  medium  or  low  S/W  criticality  used  to  make  technical 
or  managerial  decisions 

1 

1  =  Intended  use  addresses  proqram  objectives  or  analysis  that  is  not  a  siqnificant  factor 
in  the  technical  or  managerial  decision  making  process 

Criticality  Analysis:  Role  of  M&S 


•  Here’s  an  example  scheme 


Role  Level 

Definition  • 

4 

M&S  will  be  the  onlv  method  employed  to  make  a  decision 

3 

M&S  will  be  the  primary  method,  employed  with  other  non-M&S  methods 

2 

M&S  will  be  a  secondary  method,  employed  with  other  non-M&S  methods,  and  will 
provide  sianificant  data  unavailable  throuah  other  means 

1 

M&S  will  be  a  supplemental  method,  employed  with  other  non-M&S  methods,  and  will 
provide  supplemental  data  already  available  throuah  other  means 

: - W 

Criticality  Measure 


nav^a  i  R 


• Criticality  Measure  is  determined  from  level  of  reliance  on  M&S  and 
importance  of  the  decision 


• Criticality  Measure  drives  nature  and  amount  of  information  and 
effort  applied  to  W&A  of  this  model 


Importance  of 
Decisions 

Level  of  Reliance  on  M&S 

4 

3 

2 

1 

4 

4 

4  or  3 

3  or  2 

2 

3 

3 

3 

2 

2  or 

1 

2 

2 

2 

2 

1 

1 

1 

1 

1 

1 

Resources  Applied 
to  VV&A 


>  Criticality 


Source:  DD(X)  Verification,  Validation  and  Accreditation  Overview  by  Charles  Hays  of  Northrup 
Grumman  Corporation.  Presented  at  NMSO  VV&A  TWG,  Salt  Lake  City  UT  on  16  Feb  2005 


Benefit  of  the  risk-based 
VV&A  strategy 


•  Helps  you  develop  a  standard  operating  procedure  for  scoping  and  carrying 
out  W&A  efforts  on  your  program  so  that  day  to  day  implementation  is 
consistent,  effective,  efficient,  and  straightforward 

-  Upper  management  can  dictate  deviations  at  their  discretion  so  long  as  the 
deviations  and  the  rationale  are  documented 

-  Helps  you  devise  a  mechanism  for  elevating  particular  M&S  uses  to  “command 
interest”  status  for  funding  and  risk  mitigation 

•  In  the  early  stages  of  your  program,  our  W&A  approach  will  help  you  scope 
and  plan  your  W&A  strategy  over  the  life  of  the  program 

-  Get  VV&A  related  activities  in  contracts,  schedules,  budgets,  resource  planning 

•  As  the  program  progresses,  an  established  strategy  gives  you  a  way  to 
quickly  scope  the  effort  necessary  to  determine  the  credibility  of  M&S  for 
unanticipated  uses  as  the  program  evolves 


You  can  work  out  a  thoughtful  VV&A  strategy  early  on,  or  duke  it  out  on  a  case  by  case  basis 

each  time  the  issue  of  accreditation  or  credibility  comes  up. 

Why  not  think  hard  early  on  in  the  program,  and  then  get  on  with  it? 


Applying  Resources  Intelligently 


•  Other  Acquisition  programs  have  used  the  practical  methods 

-  To  determine  how  much  effort  to  put  into  VV&A  and 

-  To  get  the  most  return  on  their  investment 


•  This  method  offers  you  an  approach  for  figuring  out: 

-  What  kind  and  how  much  evidence  is  required  to  establish  a  particular 
level  of  confidence 

-  What  kind  and  how  much  evidence  is  required  to  reach  accreditation 
decision  for  particular  uses 

-  The  appropriate  level  of  review  to  generate  an  accreditation 
recommendation 

-  The  appropriate  level  of  independence  in  V&V  and  review 

-  The  appropriate  level  of  signature  authority  for  VV&A  plans  and 
reports 

-  The  appropriate  level  for  accreditation  authority 


Allot 

these 

factors 

drive 

resources 


Some  Practical  Help  with  Risk  Assessment 


•  System  Safety  community  within  DoD  and  foreign  defense  establishments 
have  grappled  with  risk  assessment 

-  Defining  qualitative  levels  of  impact  in  many  areas  (financial  loss,  political 
embarrassment,  material  loss,  personnel  loss,  etc.) 

-  Defining  qualitative  levels  of  risk  given  likelihood  and  consequence 

-  See  MIL  STD  882D  for  examples 

•  JASA  and  many  other  groups  have  a  strong  interest  in  W&A  as  risk 
reduction  and  have  contributed  to  the  literature 

-  JASA’s  Risk  Assessment  Example,  based  upon  work  we’ve  done  for  a  major 
acquisition  program,  is  an  extreme  example,  but  may  also  give  you  some  food 
for  thought  on  doing  risk  assessment  related  to  model  use 

-  See  the  DMSO  VV&A  RPG’s  core  document  “Accreditation  Agent  Role  in  VV&A 
of  Legacy  Models”  for  JASA’s  rules  of  thumb  for  what  kind  of  and  how  much 
information  is  appropriate  to  support  accreditation  assessments  given  varying 
levels  of  acceptable  risk 

•  Download  from  DMSO’s  VV&A  site:  www.vva.dmso.mil 


Questions? 


Backup  Material 


What  if  the  question  is  which  tools  to  place  emphasis  on  over  the  life  of 
the  program? 


Criticality  measure  is  one  idea 

-  Takes  into  account  role  of  M&S  in  making  decisions  and 

-  Number  and  importance  of  decisions  that  M&S  is  expected  to  support 
over  the  life  of  the  program 

-  Focus  your  efforts  on  those  M&S  that  will  be  used  most  often  for  the 
highest  profile/highest  consequence  decisions 


•  An  aid  for  tackling  how  to  best  allocate  W&A  resources  over  the  life  of  an  acquisition 
program 

•  Offered  by  the  Northrop  Grumman  team  working  with  the  DD(X)  program:  M&S 
criticality  analysis 

-  Criticality  is  a  function  of  the  dependence  on  M&S  in  making  decisions  over  the 
life  of  the  program,  and  the  nature  and  importance  of  those  decisions 

-  The  scales  used  by  the  NG  team  are  shown  on  the  next  two  slides 

•  The  idea  is  that  the  criticality  score  for  a  particular  model  can  help  determine  whether 
formal  W&A  is  required  and  how  much  effort  will  be  put  into  supporting  accreditation 

•  Interesting  idea  that  is  intuitively  appealing 

•  One  practical  implementation  issue  is  the  fact  that  the  role  of  M&S  may  differ  in 
various  phases  of  the  program  and  in  different  decisions,  so  you  might  need  a 
weighted  average  or  something 


N  A 
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Implementation  Suggestions 


•  Consider  appointing  someone  to  work  out  a  straw  man  based  upon 
the  structure  and  processes  in  place  in  your  program 
-  VV&A  person  working  in  conjunction  with  program  person  works  well 


•  Present  straw  man  to  M&S  WG  for  feedback  -  rework  incorporating 
feedback  then  present  to  MSWG  for  concurrence 


•  Once  you  have  concurrence  of  MSWG,  staff  it  up  the  chain  for 
management  approval 

•  Get  going  with  implementation  once  you’ve  got  a  solid  draft  or  you’ll 
spend  the  entire  program  arguing  about  the  nitnoids 


VV&A  is  Risk  Reduction 


Reduce  Likelihood  of  Error  =>  Reduce  Risk 


•  VERIFICATION 

-  Reduces  the  likelihood  that  the  software  you  build  (or  use)  has 
undetected  errors  that  are  fatal  to  your  intended  use 

-  Reduces  the  likelihood  that  the  data  are  inappropriate  for  the 
intended  application  or  improperly  prepared 


VALIDATION 

-  Reduces  the  likelihood  that  simulation  outputs  won’t  match  the 
“real  world”  well  enough  for  you  to  use  them  credibly  as  part  of 
the  solution  to  your  problem 

-  Reduces  the  likelihood  that  the  data  don’t  represent  the  real 
world  with  sufficient  accuracy  for  the  application 


•  ACCREDITATION 

-  Reduces  the  likelihood  that  an  inappropriate  or  unsuitable 
simulation  is  selected  for  use  in  solving  your  problem 


What’s  a  JASA 

Accreditation  Support  Package  (ASP)? 


•  A  JASA  ASP  (as  in  A-S-P,  not  the  name  of  the  snake)  is  an 
organized  way  to  document  and  relay  the  information  about  a 
model  QL  simulation  and_  its  input  data  that  is  typically  used  to 
support  an  accreditation  assessment 

-  Contents  are  based  on  the  model-related  information  elements  that 
DoD  and  Service  level  policies  either  require  or  recommend  to  support 
accreditation  decisions  and  13  years  of  experience  doing  accreditation 
support  for  DoD  acquisition  programs 


•  It  has  a  single  volume  format  organized  around  the  three  pillars  of 
M&S  credibility  conceived  by  JASA  and  adopted  by  the  Defense 
Modeling  and  Simulation  Office  (DMSO) 

-  Capability:  Does  the  simulation  do  what  you  want  it  to? 

-  Accuracy:  How  much  confidence  can  be  placed  in  the  accuracy  of 
model  results? 

-  Usability:  Is  there  enough  information/help  available  to  enable  proper, 
consistent  use  of  the  model  and  correct  interpretation  of  results? 


JASA  Accreditation  Support  Package  (ASP) 
Structure  2004  Specification 


1.0  Introduction 


Overview  of  Accreditation  Process 
Information  Needed  for  Accreditation 
Capability 
Accuracy 
Usability 


2.0  Capability 


Model  Description 
Functional  Capabilities 
Development  History 

Summary  of  Assumptions  and  Limitations 
Implications  for  Model  Use 


Software  Accuracy 
S/W  Verification  Results 
S/W  Development  and  CM  Environment 
S/W  Quality  Assessment 
Data  Accuracy 

Simulation  Data  including  Pedigree 
Data  Transformations 
Output  Accuracy 
Sensitivity  Analysis 
Benchmarking 
Face  Validation 
Results  Validation 
Implications  for  Model  Use 


4.0  Usability 


See  Accreditation  Support  Package  (ASP) 
Specification,  Joint  Accreditation  Support  Activity, 
September  2004,  Rev  B  May  2005,  JASPO-03-M-002B 


Documentation 
User  Support 
Usage  History 
Implications  for  Model  Use 


JASA’s  Evolution 


•  Predecessor  was  the  OSD-sponsored  Susceptibility  Model  Assessment 
and  Range  Test  (SMART)  Program 

-  Five  years  (FY92-96,  OSD-funded,  Tri-Service  Steering  Group) 

-  Developed  and  documented  cost  effective  VV&A  process  for  survivability  M&S 
including  Accreditation  Support  Package  (ASP)  specification 

-  Exercised  process  on  5  survivability  models 

-  Documented  processes  and  lessons  learned 

•  JASA  was  created  in  FY96  to  provide  M&S  accreditation  support  services 
to  the  larger  acguisition  community 

-  Concepts  and  processes  broadly  applicable  to  M&S  used  in  the  larger 
acquisition  community,  not  only  for  survivability 

-  Initially  under  the  auspices  of  the  Joint  Technical  Coordinating  Group  for  Aircraft 
Survivability  (JTCG/AS),  who  provided  some  infrastructure  funding  from  FY96-98 
to  assist  in  transition 

-  FY99  to  present  almost  entirely  customer  funded  with  some  specific  tasking  for 
JTCG/AS  (now  JASPO) 

-  2006  JASA  became  part  of  the  Battlespace  Simulation  &  Test  Dept  (5.4)  NAVAIR 


Terminology: 

Industry  Standards  vs.  M&S  VV&A  Policy 


Question 

SE/SysE/CMMI/ISO  9000 
Terminology 

M&S  VV&A 
Terminology 

Does  the  product  meet  the 
reauirements/specs? 

Product  Verification 

M&S  Verification  and 
Validation 

M&S  Validation  deals 
with  accuracy 
requirements 

Is  the  product  fit  for  purpose 
in  the  customer’s  intended 

environment? 

Product  Validation 

w 

M&S  Accreditation 

Accreditation  is  a 
government  decision 

What  is  the  desired  end 
state? 

r  i 

•Acceptance  by  customer  and 
payment  for  services 

•Launch  of  quality  product  or 
service 

r  1 

w 

Use  of  M&S  by  decision 
maker  with  an  acceptable 
level  of  risk 

! t! 

•Note:  CMMI  and  ISO  9000  emphasize  effective  process  rather  than  product,  but  use  of  terms  is  consistent  with  that  of  the 
Software  Engineering  (SE)  and  Systems  Engineering  (SysE)  communities 


MORE  ADVANCED 
COMBINATION  SCHEMES 

•  Useful  when  Different  Schemes  Result  In 
Different  Risk  Level  Ratings 

METHODOLOGY:  ( see  next  4  slides) 

1  .Use  Chart  #1  in  the  “Standard  Risk  Chart”  to  determine  appropriate  color:  G1 , 
Y1  or  R1 

2  2. Use  Chart  #2  in  the  “Standard  Risk  Chart”  to  determine  appropriate  color:  G2, 
Y2  or  R2 

3.  3.  Use  COMBINED  RISK  CHART  to  determine  appropriate  color:  Green, 

Yellow  or  Red. 

•  NOTE:  If  you  are  a  decision  maker  who  is  more  interested  in  very 
low  risk  (i.e.  a  Risk  Averse  Decision-maker),  use  the  COMBINED 
RISK  AVERSE  CHART  instead  of  the  COMBINED  RISK  CHART 


SAMPLE  IMPACT  TABLE 

IMPACT  MATRIX 


LEVEL  OF  RELIANCE 

IMPACT  _ 

3  2  1 


ir 


CATASTROPHIC 

5 

4 

3 

CRITICAL 

4 

3 

2 

MARGINAL 

3 

2 

1 

SAMPLE  CONSEQUENCE  TABLE 


Consequence  Matrix 

Importance  Level  of  Reliance 
of  Decision 


clook-qt- 


STANDARD  RISK  CHA 


COMBINED  RISK  CHARTS 


RISK  AVERSE  MATRIX 


NORMAL  RISK  MATRIX 
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National  Defense  Industrial  Association 

10th  Annual 

Systems  Engineering  Conference 

Track  5  M&S  Session  3B5 
10:15  am  - 12:00  pm 


Standardized  Documentation  for 
Verification,  Validation,  and 

Accreditation  - 

A  Status  Report  to  the  Systems 
Engineering  Community 

Presented  by 
DoD  M&S  Project  (DMSP) 

Project  Management  Team  (PMT) 

October  24,  2007 


Outline 


•  Introduction 

•  Background 

•  DoD  M&S  Project 

•  Concept  of  Operations 

•  Policy,  Guidance  &  Standards 

•  DoD  VV&A  Documentation  Tool 

•  Structured  VV&A  Data 

•  Summary 


A  Word  from  Our  Sponsor 


“Because  M&S  is  a  fundamental  and  essential  tool  for  acquisition 
programs,  planning  for  use  of  M&S  throughout  developmental  test  and 
evaluation  must  be  an  early  consideration  in  test  planning.  Just  as  M&S 
planning  should  be  integral  to  program  acquisition  plans  and 
systems  engineering  plans,  it  should  also  be  integral  to  the  program 
Test  and  Evaluation  Strategy  and  T&E  Master  Plan.  Important 
planning  considerations  include:  the  use  and  reuse  of  M&S 
applications  and  data  for  T&E  across  the  program  lifecycle, 
establishing  credibility  of  M&S  tools  and  data,  using  M&S  to  predict 
live  test  results,  and  using  live  test  results  to  improve  the  credibility  of 

M&S.” 


Chris  DiPetto,  Deputy  Director 
OUSD(AT&L)A&T/SSE/DT&E 

March  26,  2007 


Introduction 


•  Modeling  and  Simulation  (M&S)  is  a  key  enabler 
in  the  acquisition  process  for  systems  engineers. 


•  Using  M&S  that  provide  credible  results  is 
crucial  to  fielding  defense  weapon  systems  to 
the  warfighter. 

•  Credibility  and  confidence  in  the  use  of  M&S 
results  are  achieved  through  implementation  of 
Verification,  Validation,  and  Accreditation 
(VV&A)  processes. 

•  VV&A  is  critical  for  ensuring  M&S  is  correct,  is 
used  correctly,  and  can  produce  results  a 
systems  engineer  can  trust. 


DoD  Directive  5000.59 


•  DoD  Directive  5000.59,  DoD 
M&S  Management,  8  Aug  2007 

•  Sec  4.3:  It  is  DoD  policy  that  M&S 
management  shall ...  pursue 
common  and  cross-cutting  M&S 
tools,  data,  and  services  to 
achieve  DoD’s  goals 

•  Sec  5.1 .3.5:  The  M&S  SC  shall ... 
Oversee  ...  the  implementation  of 
best  practices  of  how  models  and 
simulations  are  effectively 
acquired,  developed,  managed, 
and  used  by  DoD  Components 
(e.g.,  verification,  validation,  and 
accreditation;  standards;  and 
protocols). 


Department  q i  Defy n^t1 

DIRECTIVE 

Nl’A  IELK 

. .  . " . .  1SSSSSSSSS^S^^ 

5-UEJECT:  Hbieiiz.:  3od  rji3iiJatcu  ;>.[■*:  Vi  Mans'catfu: 

ReScciu):  [il  D n  J  Du*-: riv t :  Kffl . 0 .  ■  TidD  }.  1  udc lin^  J  ad  fi xiui iidz.  Lt'i? j 
MitDitiue--;  "  Jinury  IS 9A  (hmby  CEDC*bd;i 
[b'J  rtopaiu  li-ill  \Eemcrnnidini]  (PER11  Docecibtj  ]4.  ' 

(i>  DoD  n  S305.1  E.  <C * t&uir&i  MimtHiMtir  Piteim- 

Ft*uw£.lP99 

MJ  SOCO  tt-M,  -PcO  ^Goditkfi  wnj  SmnljniM  Ctafxy  ' 

JjniLsn- 

(4)  rlzjLxicli  £ ; .  -;w  £sc  Ls;u:s  3 


1.  KESVJAiXS  AjQ  JUSPOS5 

Thli  Du«rto. 

l  i  ftwtu'  Rtfnron  W  m  ipdiit  policy  and  n^hMik  ftr  Pod 

riTTrHj:<-Mig.i  p*i  r.s-i  vihiLL-t  [bj 

LJ_  El: lb]::!* 5  ±it  DoI>  n-^irw  fMitj  SC'JandtriM  pcavi^DOL  cf 

F>:>:tTK<  (c>. 

L  3  AlflHUlMS  Ibt  .3f‘,  tkmimir  ftf  MJ  PtiWieaaffii  as  Hid  iAuCmi]*:,  Ifr  aulhfi  to 

KAftteai* (JJ.  DflD  5K3 


2.  A^fLjE.  Aj-IutV 

2A.  TI35  Dinscvt  jppj&irn  dje  QS:«  c:  ±t  S«zituy  of D«5eut  zaeMabury 
L-ultu^l  n:  die  Jai^i  *.  Lefs  af  d&e  Tebr  kidf.  tbe  Cozcbnio 
Ccizm^uda.  Cf^st  oJtie  lnJpfr:5W  ■ieaenicftie  Sepa^zaeiT -pf  Dc5k.w=  Defeat 

Askkba*  £ifc  I>dD  7i^d  AflLlHa.  and  £1  GLksfc  ra  falaisAticaa]  eutiLAe-  williia  b* 
^DkTst&a  fliarejSej  w  u  me  "EaE 


1  Amirtit  itxi  C  M=:i"  i  j*4  Svp.v?n=b 


DoD  Instruction  5000.61 


•  DoD  Instruction  5000.61 , 
DoD  M&S  VV&A,  1 3  May 
2003 

•  Sec  4.1 :  It  is  DoD  policy 
that ...  Models  and 
simulations  used  to  support 
major  DoD  decision-making 
organizations  and 
processes  ...  shall  be 
accredited  for  that  specific 
purpose 


Department  of  Defense 

INSTRUCTION 


NUMBER  5000.6I 
Vtay  13.  n:i:n 


U9H(A12L) 


SUEJEC  F:  DoD  Modeling  and  Simulation  (M&S)  Verification,  VabifatiniL,  and 
Accradilation  (VrV£_-\;i 

lie  loroiic  v *:  ■  a  i  Du  D  I  u*Lnn:  I  imi  500(  1.6 1 , "  I  3oD  MuiL-  li  no  and  S  imu  kiLim  ■  M Ji  S  i 
Verification,  Validation,  and  Accreditation  fWiA]" April  29, 199(5 
(hereby  canceled) 
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This  TnaLncti  oa: 

1.1 .  Reissues,  rc fcrcncc  (a)  Lo  implement  policy,  amign  responsibilities.  and 
prescribe  procedures  under  rafarancs  (b)  I  or  Lin-  verification.  valiiition,  and 
accreditati  on  i  VV&A  i  of  DoD  mode  In  and  simulations  and  Llu  ir  BaacxdartBd  diila. 

1 .2 .  A  uthori  me  puhl  i cation  o f  I  3ol  3  5000.6 1  -G, '  I  3ol  3  Veri  lien  I  ion ,  Va  lid.i  Lion,  and 
Accra  dilation  GLridaJ"catmEteut  v  ith  DoD  5025. 1-M  i  rc  Terence  ( c)) . 


2.  AEH  K  Ami -ITV  ANI3  STOP  E 


I'll lihdmcliou  applies  to: 


Sec  6.4:  VV&A  information 
shall  be  documented 


Need 


•  Since  1996  organizations  DoD-wide  have 
been  implementing  VV&A  processes  and 
capturing  VV&A  information 


•  Plethora  of  DoD-,  Service-,  and 
organization-level  documentation 
guidance 

-  No  consistency  in  formats  and  content 
descriptions 

•  No  easy  method  to  identify  published 
VV&A  information 


|}|  Importance  of  VV&A  Information 

Documenting  VV&A  information  consistently 
across  DoD  yields  many  returns  including  the 
capability  to  share  that  information  with  future 
users  of  M&S. 

VV&A  information  tells  a  potential  user  about 

-  M&S  assumptions  (simplifications  and  potential  failure 
points) 

-  M&S  capabilities  (what  the  M&S  can  be  used  to  do) 

-  M&S  limitations  (what  it  should  not  be  used  to  do) 

VV&A  information  saves  potential  users  time 
and  money  finding  an  M&S  that  satisfies  or 
partially  satisfies  their  needs  to  use  M&S. 


Background 


•  In  2005,  a  DoD-sponsored  Tri-Service  VV&A  Templates  Tiger  Team 
developed  templates  for  four  core  VV&A  documents: 

-  Accreditation  Plan 

-  V&V  Plan 

-  V&V  Report 

-  Accreditation  Report 

•  Purpose  was  to  enable  expanded  M&S  reuse  by  building  foundation 
for  consistent  V&V  information  to  support  accreditation  decisions. 

•  Templates  resulted  in  draft  DoD  Standard  Practice 

-  M&S  VV&A  Documentation  Templates  (MIL-STD-XXX002) 

-  provides  common  framework  for  sharing  information  throughout  VV&A 
processes 

-  using  templates  helps  users  better  understand  if  M&S  can  meet  their 
needs 

-  templates  make  it  easy  to  know 

•  what  kind  of  information  is  available 

•  where  to  look  in  the  document  for  that  information 

•  Templates  automated  by  DoD  VV&A  Documentation  Tool  (DVDT) 


MIL-STD-XXX002  (draft) 


•  MIL-STD-XXX002  (draft)  DoD 
Standard  Practice  “M&S  VV&A 
Documentation  Templates” 

•  Specifies  procedures  on 
documenting  information 
obtained  through  implementing 
the  VV&A  processes  for  M&S 
when  their  outputs  will  be  used 
to  supplement  decision  making 
in  DoD. 

•  May  be  cited  as  solicitation 
requirements. 

•  Guidance  should  be  applied  in 
accordance  with  the  scope  of 
the  specific  purpose  for  using 
M&S. 
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DoD  M&S  Project 


•  Sponsor:  Department  of  Defense  (DoD) 

M&S  Steering  Committee 
(M&S  SC) 

•  Oversight:  Acquisition  Community  Lead 


•  Project  title:  Standardized  Documentation 

for  VV&A 


Project  Management  Structure 


Project  Scope 


•  Three  major  tasks  and  associated 
deliverables: 

-  recommend  updates  to  associated  policy, 
guidance,  and  standards  documents 

-  develop  VV&A  XML  schema  and  VV&A 
ontology  for  M&S 

-  produce  DVDT 


M&S  Practices  Gaps 


•  Address  gaps  documented  in  M&S  SC  Common 
and  Cross-Cutting  Business  Plan 

•  REUSE 


-  Potential  users  find  it  difficult  to 

•  locate,  access,  and  assess  M&S  resources  and  to  identify 
potential  reuse  candidates 

•  clearly  understand  the  capabilities  of  candidate  model  and 
simulation  resources 

•  assess  the  difference  between  the  functionality  of  reuse 
candidates  and  the  capabilities  that  are  needed 

•  VV&A 

-  There  is  no  mature  method  for  deriving  VV&A  costs 

-  Standardized  VV&A  documentation  templates  are 
needed 


Acquisition  Objectives  &  Actions 


Address  objectives  and  actions  identified  in  the 

DoD  Acquisition  M&S  Master  Plan 

OBJECTIVES 

-  Obj  2:  Enhance  the  technical  framework  for  M&S 

-  Obj  4:  Improve  M&S  use 

ACTIONS 

-  Establish  a  standard  template  of  key  characteristics 
(metadata)  to  describe  reusable  M&S  resources 

-  Enhance  the  means  (e.g.,  directory  service,  registries, 
bulletin  boards)  to  discover  the  existence  of  reusable 
resources  required  for  M&S  and  contact  information 

-  Require  standardized  documentation  of  VV&A  DoD- 
wide 


Concept  of  Operations 


DoD  VV&A 
Documentation 
Tool  Domain 


l  DoD  Discovery  * 

^tadata  Specifica11 

^■oballnformation  Grid  Service  -Oriented  Architecture  Domaii 


Policy,  Guidance  &  Standards 


M&S 

Level  1  ^ 

DoDI  5000.59 


Level  2 


\L 


Acquisition 

I 

DoDI  5000.2 


Data 

I 

DoD  Discovery 
Metadata 

Specification  (DDMS) 


VV&A  information  is  important  not  only  for  the  decision  at  hand, 

but  for  future  decisions  to  reuse  M&S 


Policy,  Guidance  &  Standards 

•  Develop  recommended  changes  to  policy, 
guidance,  and  standards  documents  to 
advocate : 

-  making  the  standardized  templates  a  MIL-STD 

-  using  standardized  templates  for  documenting  VV&A 

-  automating  production  of  VV&A  information 

•  Provide  recommendations  to  Acquisition  M&S 
Working  Group 

•  Deliver  recommendations  to  Acquisition  member 
of  M&S  Steering  Committee  (M&S  SC): 

-  formally  request  changes  to  related  policy,  guidance, 
and  standards  documents  through  Communities 


Policy,  Guidance  &  Standards 
Recommendations 


M&S  Steering 
Committee 


AMSWG  & 


DoD  M&S  Project  M&S  IPT 
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Producer  /  Consumer  View 


DoD  VV&A  Documentation  Tool 

(DVDT) 

•  Technology  development  effort  to  automate  standard 
DoD  VV&A  templates 

•  Benefits  to  automation 

-  expedites  VV&A  documentation  production  process 

-  ensures  standardization  of  content  and  format  across  DoD 

-  ensures  compliance  with  policy  and  guidance 

-  guides  Producer  through  the  VV&A  processes 

-  enables  content  consistency  and  completeness  across  all 
Communities 

-  facilitates  and  contributes  to  M&S  reuse 

-  provides  quality  and  complete  VV&A  information  to  stakeholders 
faced  with  making  decisions  on  the  application  of  M&S  results 

-  provide  standardized  methods  to  communicate  VV&A 
information  at  appropriate  levels  of  detail 

-  ensures  appropriate/useful  metadata  is  extracted  and  posted 
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Structured  VV&A  Documents 


Effective  data  sharing  requires  the 
commonly  understood  representation  of 
the  data 

Defined  and  published  data  structures 
facilitate  information  exchange  and 
application  development 


VV&A  Documents 
(Structured  Data  and  Semantics) 


Producer 


User  employs  the 
DVDT  to  create 
VV&A  documents 
stored  as  XML  files 


Structured  Data  stored  in  XML  files 
and  conforms  to  established  XML 
schemas 
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XML  data  structure  and 
content  conforms  to  the 
DVDT  schema 
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Auto-Generation  of  Documentation 
from  stored  XML  data 


Producer 


User  opens  VV&A 
documents  in  DVDT 
and  generates 
products  in  desired 
formats  (e.g.,  XML, 
HTML,  PDF,  DOC) 
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Metadata  Design  Considerations 


•  Formalize  structure  and  content  of  standard  VV&A  document 
templates  into  XML  schemas 

•  Capture  key  elements  from  the  standard  VV&A  document 
templates  for  posting  to  the  M&S  resource  registry 

•  Include  mandatory  DoD  Discovery  Metadata  Specification 
(DDMS)  elements  sufficient  to  construct  valid  DDMS 
Resource  metadata  document 

•  Include  mandatory  DoD  M&S  Community  of  Interest 
Discovery  Metadata  Specification  elements 

•  Reuse  existing  XML  namespaces  directly  or  through 
transformations  (mappings) 

•  Comply  with  best  practice  as  described  in  XML  Naming  and 
Design  Rules  (e.g.,  DoN,  UN/CEFACT) 


M&S  COI  Discovery  Metadata 
Specification:  Organization  Structure 


Specific  Community 
M&S  Datasets 


Specification  work  occurring  in  parallel  with  DVDT  schema  development 
offering  opportunity  to  maximize  compatibility  and  mutual  benefit. 


VV&A  Metadata 

(Structured  Data  and  Semantics) 


DVDT  generates  VV&A  metadata  for 
posting  to  M&S  Resource  Registry 
Domain  to  support  search  in  the  GIG 
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Structured  VV&A  Metadata 


•  Consistent  use  of  the  DVDT  across  the 
DoD  and  Military  Departments,  will 

-  enable  publishing  of  VV&A  metadata 

-facilitate  discovery  and  sharing  of  VV&A 
information  over  the  Global  Information  Grid 
(GIG) 

•  Align  metadata  design  with  DoD  Discovery 
Metadata  Specification  (DDMS)  and  M&S 
Community  of  Interest  Discovery  Metadata 
Specification 


VV& A  Ontology  for  M&S 


•  Establishes  technical  case  for  application  of 
formalized  semantics  relating  to  VV&A 
processes  and  records 

•  Employed  by  software  and  humans  to  discover 
M&S  resources  and  to  assess  suitability  for  the 
intended  use 

•  Will  use  Web  Ontology  Language  (OWL)  and 
other  Semantic  Web  standards 
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For  more  information, 
to  become  a  DVDT  beta  tester,  or 
to  use  the  DVDT  to  support  a  VV&A  project, 
contact  any  one  of  the  authors  below. 
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ABSTRACT:  Using  models  and  simulations  that  provide  credible  results  in  the  systems  engineering  process  is 
crucial  to  fielding  defense  weapon  systems  more  effectively  to  the  warfighter.  Employing  distributed,  live-virtual- 
constructive  synthetic  environments  that  produce  results  that  can  be  used  with  confidence  is  essential  to  support 
development  and  testing  of  interoperable  systems  for  joint  capabilities.  Credibility  and  confidence  in  the  use  of 
modeling  and  simulation  (M&S)  results  can  be  achieved  only  through  the  implementation  of  standard  Verification, 
Validation,  and  Accreditation  (VV&A)  processes.  M&S  is  a  key  enabler  for  systems  engineers  throughout  the 
acquisition  process.  VV&A  is  critical  for  ensuring  an  M&S  is  correct,  is  used  correctly,  and  can  produce  results  a 
systems  engineer  can  trust. 

The  Department  of  Defense  (DoD)  Modeling  and  Simulation  Steering  Committee  (M&S  SC)  Acquisition  M&S 
Community  Lead,  Mr.  Chris  DiPetto,  Deputy  Director  for  Developmental  Test  and  Evaluation,  is  sponsoring  several 
Acquisition  M&S  Projects.  One  of  those  projects  is  titled,  "Standardized  Documentation  for  Verification,  Validation, 
and  Accreditation.  ”  This  paper  will  update  the  Systems  Engineering  Community  on  what  the  project  is  about  and 
progress  that  has  been  made.  It  will  provide  information  on  the  development  of  standardized  content  and  format 
requirements  for  four  core  VV&A  documents,  the  technology  development  efforts  to  automate  those  templates  to 
ensure  standardization  across  the  DoD  and  Military  Departments,  and  the  work  underway  to  identify  VV&A  metadata 
that  will  enable  the  sharing  of  information  across  all  M&S  Communities  via  the  Global  Information  Grid  anywhere  in 
the  world  and  at  anytime.  Additionally,  the  paper  will  identify  gaps  from  the  M&S  SC  Common  and  Cross-Cutting 
Business  Plan  and  objectives  from  the  DoD  Acquisition  M&S  Master  Plan  that  are  being  addressed  by  this  project. 
Finally,  the  paper  will  provide  an  overview  of  the  project  including  scope,  schedule,  and  deliverables. 


1.  Introduction 

The  Department  of  Defense  (DoD)  Modeling  and 
Simulation  Steering  Committee  (M&S  SC)  Acquisition 
M&S  Community  Lead,  Mr.  Chris  DiPetto,  Deputy 
Director  for  Developmental  Test  and  Evaluation,  is 
sponsoring  several  Acquisition  Modeling  and  Simulation 
(M&S)  Projects.  One  of  those  projects  is  titled, 
’’Standardized  Documentation  for  Verification, 
Validation,  and  Accreditation  (VV&A).”  This  paper 
updates  the  Systems  Engineering  Community  on  what  the 
project  is  about  and  progress  that  has  been  made.  It 
provides  information  on  the  development  of  standardized 
content  and  format  requirements  (i.e.,  templates)  for  four 
core  VV&A  documents,  the  technology  development 
efforts  to  automate  those  templates  to  ensure 
standardization  across  the  DoD  and  Military  Departments, 
and  the  work  underway  to  identify  VV&A  metadata  that 
will  enable  the  sharing  of  information  across  all 
Communities  enabled  by  M&S  via  the  Global 
Information  Grid  (GIG)  anywhere  in  the  world  and  at 
anytime.  Additionally,  the  paper  identifies  gaps  from  the 
M&S  SC  Common  and  Cross-Cutting  Business  Plan  and 
objectives  from  the  DoD  Acquisition  M&S  Master  Plan 
that  are  being  addressed  by  this  project.  Finally,  it 
provides  an  overview  of  the  project  including  scope, 
schedule,  and  deliverables. 

Using  models  and  simulations  that  provide  credible 
results  in  the  systems  engineering  process  is  crucial  to 
fielding  defense  weapon  systems  more  effectively  to  the 
warfighter.  Employing  distributed,  live-virtual- 
constructive  synthetic  environments  that  produce  results 
that  can  be  used  with  confidence  is  essential  to  support 
development  and  testing  of  interoperable  systems  for  joint 
capabilities.  Confidence  in  the  use  of  M&S  results  can  be 
achieved  only  through  the  implementation  of  standard 
VV&A  processes  that  are  understood  and  employed  by 
the  M&S  communities.  M&S  is  a  key  enabler  for  systems 
engineers  throughout  the  acquisition  process.  VV&A  is 
critical  for  ensuring  an  M&S  is  correct,  is  used  correctly, 
and  can  produce  results  a  systems  engineer  can  trust. 

2.  Background 

DoD  Instruction  (DoDI)  5000.61  [1]  sets  policy  requiring 
accreditation  of  all  models  and  simulations  “used  to 
support  major  DoD  decision-making  organizations  and 
processes”  and  all  models  and  simulations  “used  to 
support  joint  training  and  joint  exercises.”  The  Instruction 
requires  DoD  components  to  “establish  VV&A  policies 
and  procedures  for  models  and  simulations  they  develop, 
use,  or  manage.”  Moreover,  the  Instruction  requires 
Principal  Staff  Assistants  and  heads  of  the  DoD 
Components  to  hold  M&S  proponents  accountable  and 


responsible  for  “verification  and  validation  of  their 
assigned  M&S,  as  well  as  the  documentation  of  those 
activities,”  and  to  hold  individual  data  producers 
accountable  and  responsible  for  “the  quality  of  their  data 
or  data  products  provided  for  M&S  use.” 

Since  1996  when  DoDI  5000.61  [1]  was  first 

promulgated,  organizations  DoD-wide  have  been 
attempting  to  implement  VV&A  processes  and  capture 
VV&A  information.  Over  the  years,  guidance  for 
implementing  VV&A  was  provided  in  the  form  of 
Service-  and  organizational-level  instructions, 
recommended  practices,  guidebooks,  handbooks,  and 
standards.  The  requirements  for  documenting  VV&A 
information  varied  from  Service-to-Service,  organization- 
to-organization,  and  community-to-community,  but 
generally  all  required  the  same  types  of  information 
needed  to  gain  confidence  in  the  application  of  M&S 
results  for  an  intended  use.  Because  there  were  common 
general  requirements,  the  Service  VV&A  representatives 
came  together  in  2005  as  part  of  a  DoD-sponsored  VV&A 
Templates  Tiger  Team  to  begin  work  on  developing  one 
set  of  templates  for  four  core  VV&A  documents:  the 
Accreditation  Plan,  V&V  Plan,  V&V  Report,  and  the 
Accreditation  Report.  The  purpose  was  to  enable 
expanded  M&S  reuse  by  building  the  foundation  for 
consistent  V&V  information  to  support  accreditation 
decisions.  The  result  of  that  effort  will  be  a  DoD  Standard 
Practice  (draft  MIL-STD-XXX002)  [2]  that  provides  a 
common  framework  for  the  sharing  of  information 
throughout  the  VV&A  process.  The  templates  captured  in 
the  standard  practice  will  be  automated  by  the  DoD 
VV&A  Documentation  Tool  (DVDT).  Using  templates 
with  standard  format  and  content  requirements  to 
document  VV&A  information  across  DoD  will  help  users 
better  understand  if  an  M&S  can  meet  their  needs  because 
they  will  know  what  kind  of  information  is  available  and 
where  to  look  in  the  document  for  that  information. 

The  DVDT  is  the  latest  tool  to  address  the  need  to  capture 
VV&A  information  in  a  consistent  format  with  consistent 
content.  Prior  prototype  versions  used  by  various 
organizations  across  DoD,  preceded  the  DVDT  and 
provided  the  baseline  for  functional  requirements  [3].  The 
DVDT  will  be  discussed  more  in  Section  4. 

2.1  Sharing  VV&A  Information 

The  primary  product  of  the  VV&A  processes  is 

information.  [4] 

Documenting  VV&A  information  consistently  across 
DoD  will  yield  many  returns,  one  of  which  is  the 
capability  to  share  that  information  with  future  users  of 
M&S.  VV&A  information  can  tell  a  potential  user  about 
the  M&S  assumptions,  capabilities,  and  limitations.  It 


provides  a  description  of  what  the  M&S  can  be  used  to  do 
(capabilities)  and  also  what  it  should  not  be  used  to  do 
(limitations).  This  information  can  save  time  and  money 
for  potential  users  if  they  can  find  a  match  that  satisfies  or 
partially  satisfies  their  needs  to  use  M&S. 

Using  standardized  terminology  will  make  VV&A 
information  easier  to  discover  and  share  over  the 
GIG.  [5] 

An  ontology  defines  a  common  vocabulary  and  a  shared 
understanding  that  enables  reuse  of  knowledge.  A 
common  VV&A  vocabulary  will  enable  the  development 
of  a  standard  XML  schema  that  will  facilitate  the  sharing 
of  VV&A  information  in  GIG  service-oriented  and  net- 
centric  architectures. 

Warfighters  around  the  world  depend  on  M&S  and  need 
ready  access  to  VV&A  information  that  can  provide  them 
the  basis  for  using  M&S  results  to  inform  decisions.  It  is 
Joint  Staff  policy  to  assure  that  all  information  technology 
(e.g.,  M&S)  that  is  used  to  support  operations  meet 
interoperability  requirements  and  are  supportable  over  the 
GIG  [6].  Section  5  will  discuss  these  efforts  in  more 
detail. 

2.2  Gaps,  Objectives,  and  Actions 

This  DoD  M&S  Project  (discussed  more  in  Section  3) 
addresses  gaps  affecting  the  effective  use  of  M&S 
throughout  DoD  identified  in  the  M&S  SC  Common  and 
Cross-Cutting  Business  Plan  [7].  That  business  plan 
captures  in  one  place  corporate  level  M&S  requirements 
and  needed  capabilities.  The  gaps  are  segregated  into  one 
of  three  sections  —  Technology,  M&S  Practices,  and 
Representations. 

The  M&S  Practices  area  addresses  the  guidance,  business 
rules,  and  information  exchange  mechanisms  that  support 
planning,  development,  and  use  of  M&S.  Reuse  and 
VV&A  are  identified  as  M&S  Practices  gap  areas: 

REUSE 

Potential  users  find  it  difficult  to: 

—  locate,  access,  and  assess  M&S  resources  and  to 
identify  potential  reuse  candidates, 

—  clearly  understand  the  capabilities  of  candidate  model 
and  simulation  resources,  and 

—  to  assess  the  difference  between  the  functionality  of 
reuse  candidates  and  the  capabilities  that  are  needed. 

VV&A 

—  There  is  no  mature  method  for  deriving  VV&A  costs. 

—  Standardized  VV&A  documentation  templates  are 
needed 


The  DoD  M&S  Project  addresses  these  gaps.  In  addition, 
because  the  DoD  M&S  Project  is  sponsored  by  the 
Acquisition  Community  Lead,  it  also  addresses  the 
following  objectives  and  actions  identified  within  the 
DoD  Acquisition  M&S  Master  Plan  [8]. 

OBJECTIVES 

—  Enhance  the  technical  framework  for  M&S. 

—  Improve  M&S  use. 

ACTIONS 

—  Establish  a  standard  template  of  key  characteristics 
(metadata)  to  describe  reusable  M&S  resources. 

—  Enhance  the  means  (e.g.,  directory  service,  registries, 
bulletin  boards)  to  discover  the  existence  of  reusable 
resources  required  for  M&S  and  contact  information. 

—  Require  standardized  documentation  of  VV&A  DoD- 
wide. 

The  Master  Plan  documents  the  actions  needed  to 
improve  M&S  support  to  the  DoD  acquisition  process. 
The  specific  actions  defined  within  the  plan  will  foster 
better  tools  and  processes  to  support  systems  engineering, 
acquisition  decision  making,  development  of  joint 
capabilities,  and  realization  of  cost  efficiencies  [8]. 

3.  DoD  M&S  Project 

The  M&S  SC  established  several  DoD  M&S  Projects  in 
FY07  through  the  proposal  process  depicted  in  Figure  1 
and  described  below. 
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Figure  1.  M&S  SC  Proposal  Process 

Develop  Project  Proposals  —  Communities  led  the 
development  of  and  submittal  of  proposals  for  projects 
addressing  gaps  in  the  M&S  SC  Common  and  Cross- 
Cutting  Business  Plan  [7]. 

Develop  Project  Portfolios  —  The  M&S  Integrated 
Product  Team  (IPT)  and  the  Portfolio  Development 
Group  assessed  the  project  proposals  and  identified 
promising  proposals  that  addressed  the  gaps.  These  two 
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groups  then  worked  with  the  proposal  submitters  to 
improve  the  proposal's  ability  to  address  gaps. 

Select  Projects  —  The  M&S  IPT  evaluated  the  proposals 
and  made  selection  recommendations  to  the  M&S  SC. 
The  M&S  SC  decided  which  proposals  to  fund. 

This  project  was  vetted  successfully  through  that  process, 
found  to  address  several  gaps  and  acquisition  objectives, 
and  selected  for  funding  in  FY07  with  a  period  of 
performance  through  September  2008. 

3.1  Acquisition  Governance 

The  Acquisition  Community  M&S  SC  member  provides 
oversight  of  the  project  through  the  Acquisition  member 
of  the  M&S  IPT.  Figure  2  depicts  the  DoD  acquisition 
governance  structure  [9]. 


Figure  2.  DoD  Acquisition  Governance  Structure 


The  Acquisition  M&S  Working  Group  (AMSWG)  is 
chartered  by  the  DoD  Systems  Engineering  Forum  to 
assist  program  managers  and  acquisition  professionals  by 
improving  the  utility  of  M&S  in  the  acquisition  of  defense 
capabilities.  In  this  capacity,  the  AMSWG  addresses 
common  concerns,  aligns  technical  initiatives,  and 
pursues  cross-cutting  issue  resolution  [10].  Representing 
the  interests  of  the  acquisition  M&S  community,  the 
AMSWG  also  acts  as  the  sounding  board  for  this  project 
by  providing  guidance  and  direction  with  respect  to 
requirements  for  the  various  tasks  and  deliverables. 

3.2  Project  Management 

The  team  is  led  by  the  Space  and  Naval  Warfare  Systems 
Center  (SPAWARSYSCEN)  in  Charleston,  South 
Carolina,  and  its  structure  is  depicted  below  in  Figure  3. 


Figure  3.  Project  Management  Structure 

The  Project  Management  Team  and  Architecture  & 
Software  Development  Team  both  are  led  by 
SPAWARSYSCEN  Charleston.  The  Taxonomy  & 
Metadata  Team  is  led  by  the  Naval  Postgraduate  School. 

3.3  Scope 

The  project  has  three  major  tasks  and  associated 
deliverables: 

—  Produce  the  DVDT 

—  Develop  a  VV&A  XML  schema  and  VV&A 
ontology  for  M&S 

—  Recommend  updates  to  associated  policy,  guidance, 
and  standards  documents 

The  purpose  of  the  project  is  to  support  the  various  DoD- 
and  Service-level  communities  by  delivering  a  tool  that 
produces  standardized  VV&A  documentation  and  a 
VV&A  XML  schema  that  meets  net-centric  architecture 
requirements  for  sharing,  discovering,  and  retrieving 
VV&A  information  within  the  GIG  enterprise.  The 
project  is  also  working  towards  incorporating  references 
to  the  standard  practice,  DVDT,  and  VV&A  metadata  into 
the  appropriate  M&S,  data,  and  acquisition  policy  and 
guidance  documents. 

The  beta  versions  of  the  DVDT  and  supporting  XML 
schema  are  expected  in  the  first  quarter  of  FY08.  The 
final  versions  of  the  tool,  XML  schema,  and  VV&A 
ontology  are  scheduled  for  fourth  quarter  of  FY08. 


3.4  Concept  of  Operations 


Figure  4.  High  Level  Concept  of  Operations 


Figure  4  presents  the  high-level  concept  of  operations 
for  how  the  three  major  deliverables  support  the  M&S 
communities. 


Policy,  guidance,  and  standards  in  the  areas  of  M&S, 
acquisition,  and  data  (depicted  in  Figure  5)  form  the 
foundation  for  everything  to  work  together. 


Ensuring  consistent  use  of  the  DVDT  across  the  DoD 
and  Military  Departments,  will  enable  publishing  of 
VV&A  metadata,  which,  in  turn,  will  facilitate  the 
discovery  and  sharing  of  VV&A  information  over  the 
GIG.  The  new  DoDD  5000.59  [11]  was  signed  in 
August  2007  and  states  that  "M&S  management  shall 
. . .  pursue  common  and  cross-cutting  M&S  tools,  data, 
and  services."  Additionally,  it  provides  direction  to  the 
M&S  SC  to  "oversee  ...  the  implementation  of  best 
practices  of  how  models  and  simulations  are  effectively 
acquired,  developed,  managed,  and  used  by  DoD 
Components  (e.g.,  verification,  validation,  and 
accreditation;  standards,  and  protocols)."  Based  upon 
the  specificity  of  the  published  directive,  a  new 
instruction  (DoDI  5000.59)  may  well  be  needed  to 
implement  the  provisions  in  the  directive.  Along  with 
DoDD  5000.59  [11],  the  new  instruction  will  be 
important  for  all  matters  related  to  M&S  across  DoD. 
DoDI  5000.2  [12]  is  key  to  acquisition  procedures  for 
using  M&S.  Affecting  change  to  these  two  important 
level- 1  documents  will  be  worked  through  the 
appropriate  channels. 


Figure  5.  Policy,  Guidance  &  Standards 


Another  important  aspect  of  this  task  is  to  affect 
changes  in  level-2  policy,  guidance,  and  standards 
documents.  The  level-2  M&S  documents  include,  the 
draft  DoD  Standard  Practice  (MIL-STD-XXX002)  [2], 
DoD  5000.61  [1],  and  the  online  VV&A 

Recommended  Practices  Guide  [13].  The  level-2 
acquisition  documents  include  the  Defense  Acquisition 
Guidebook  (focusing  particularly  on  Section  4.5.7 
Modeling  and  Simulation)  [14],  and  online  Continuous 
Learning  Modules  provided  by  the  Defense 
Acquisition  University’s  Continuous  Learning  Center 
[15],  focusing  on  these  two  modules  in  particular: 

—  CLE  Oil  M&S  for  Systems  Engineering 

—  CLE  023  M&S  for  Test  &  Evaluation 

The  project  also  will  recommend  updating  the  DoD 
Discovery  Metadata  Specification  (DDMS)  with  the 
VV&A  metadata  for  general  use  across  the  enterprise. 

The  M&S  SC  members  represent  the  driving  forces 
behind  making  the  necessary  changes  to  these  various 
policy,  guidance,  and  standards.  Together  they 
represent  a  unified  front  for  M&S  management  across 
DoD. 

The  concept  of  operations  in  Figure  4  starts  with  a 
consumer’s  need  to  use  M&S.  The  consumer  employs 
the  GIG  to  conduct  a  semantic  search  for  VV&A 
information  to  locate  resources  that  best  meet 
requirements  for  the  use  of  M&S.  VV&A  metadata 
transferred  from  the  DVDT  will  be  searchable  in  the 
M&S  Resource  Domain.  Based  upon  the  information 
retrieved  from  the  M&S  Resource  Registry  Domain, 
the  consumer  is  exposed  to  information  that  can  inform 
the  decision  to  reuse  a  legacy  M&S  ”as  is,”  to  modify  a 
legacy  M&S,  or  to  build  a  new  M&S.  The  producer 
uses  the  DVDT  to  document  VV&A  planning, 
implementation,  and  reporting.  The  DVDT  uses  XML 
source  data  to  produce  printable  documents.  When  the 
producer  initiates  a  VV&A  project  in  the  DVDT, 
VV&A  metadata  will  be  made  available  to  the  M&S 
Resource  Registry  Domain.  When  a  VV&A  document 
is  finalized  and  approved,  additional  VV&A  metadata 
will  be  made  available  to  the  M&S  Resource  Registry 
Domain. 

4.  DoD  VV&A  Documentation  Tool 

The  DVDT  is  a  technology  development  effort  to 
automate  the  standard  DoD  VV&A  templates  that  are 
captured  in  the  DoD  Standard  Practice  [2].  Automation 
of  the  templates  will  save  users  time  by  expediting  the 
VV&A  documentation  production  process  and  will 
ensure  standardization  of  content  and  format  across 


DoD  and  the  Military  Departments.  Additionally, 
automation  provides  several  other  benefits: 

—  Ensure  compliance  with  policy  and  guidance 

—  Guide  users  through  the  VV&A  process 

—  Enable  content  consistency  and  completeness 
across  all  Communities  enabled  by  M&S 

—  Facilitate  and  contribute  to  M&S  reuse 

—  Provide  quality  and  complete  VV&A  information 
to  stakeholders  faced  with  making  decisions  on  the 
application  of  M&S  results 

—  Provide  standardized  methods  to  communicate 
VV&A  information  at  appropriate  levels  of  detail 

When  the  work  to  develop  the  standard  templates  was 
completed,  efforts  turned  to  identifying  requirements 
for  a  DoD  tool  to  automate  the  production  of  VV&A 
documents.  Initially  the  requirements  effort  was  led  by 
the  M&S  Coordination  Office  (M&S  CO)  and  now  is 
part  of  this  project. 

Because  the  DVDT  is  a  tool  for  use  across  DoD  and 
the  Military  Departments,  requirements  for  the  tool 
reflect  the  needs  of  a  broad  population  that  cuts  across 
all  communities  enabled  by  M&S.  Examples  of  several 
high-level  functional  requirements  include: 

—  web-enabled  with  Secure  Socket  Layer 

—  Common  Access  Card  or  Public  Key  Infrastructure 
certificate  access  to  tool  (for  Government, 
Military,  Civilian,  and  Contractors) 

—  private  VV&A  project  management 

—  VV&A  project  owners  grant  access  permission  to 
other  VV&A  Team  members 

—  log  of  users 

—  log  of  changes  made  to  documents 

—  secure  data  storage 

—  requirements  traceability  across  all  documents 

—  produce  four  different  documents 

—  common  information  update  across  all  documents 

—  M&S  Word-compatible,  PDF  and  HTML  formats 

—  automatic  numbering  of  sections  and  subsections 

—  use  of  bold,  italics,  numbered  and/or  bulleted  lists 

—  capability  to  insert  graphics,  images,  and  tables 

—  capability  to  send  metadata  about  final  approved 
documents  to  the  M&S  Resource  Registry 
Domain. 

Additionally,  the  DVDT  offers  a  flexible  experience. 
User  will  choose  between  a  guided  wizard  interface 
and  a  ”what  you  see  is  what  you  get”  display.  The  tool 
also  offers  a  context-sensitive  help  system  that  includes 
links  to  the  appropriate  policy  and  guidance 
documents. 

The  DVDT  is  being  developed  in  conjunction  with  and 
will  be  compliant  with  the  VV&A  XML  schema  and 
VV&A  ontology  for  M&S  described  in  Section  5. 


5.  Structured  VV&A  Data 

Effective  data  sharing  requires  a  commonly  understood 
representation  of  the  data.  In  a  web-based,  network¬ 
centric  architecture,  data  sharing  through  common 
representation  is  facilitated  through  the  use  of  the 
Extensible  Markup  Language  (XML)  [16]  and  XML 
Schema  language  [17]  standards.  The  XML  Schema 
language  is  used  to  define  the  data  structure  and  valid 
content  of  XML  documents.  Design  and  development 
of  the  DVDT  includes  design  of  an  XML  Schema 
description  of  information  contained  in  the  Defense 
Standard  Practice  for  VV&A  documentation  [2].  The 
schema  will  describe  data  types  and  constraints  suitable 
for  ensuring  the  compliance  of  the  XML  documents 
created  by  the  DVDT.  The  DVDT  will  be  designed  to 
read  and  write  XML  instance  documents  that  validate1 
against  this  schema.  Selected  metadata  about  particular 
VV&A  resources  entered  using  the  DVDT  (resulting  in 
XML  instance  documents)  will  be  available  to  the 
M&S  Resource  Registry  Domain. 

Previous  work  on  the  prototype  VV&A  documentation 
tool  is  being  leveraged  to  define  the  XML  structures 
and  content  for  the  current  project.  The  DVDT  XML 
schema  will  be  developed  in  accordance  with  current 
Department  of  the  Navy  XML  Naming  and  Design 
Rules  [18].  To  promote  visibility  of  this  structural 
metadata,  the  schema  will  be  posted  to  the  DoD 
Metadata  Repository  for  community  reference  and  use. 
To  be  responsive  to  user  needs,  the  project  will 
coordinate  with  GIG  M&S  Community  of  Interest 
(COI)  activities  defining  standards  and  best  practices 
for  metadata,  data  mediation  and  services,  relating  this 
work  to  VV&A  processes,  data  and  services  as 
applicable. 

The  following  paragraphs  provide  additional 
information  about  data  sharing  requirements  in  the  GIG 
architecture.  The  Standardized  Documentation  for 
VV&A  project  will  comply  with  data  sharing  policies 
for  widest  possible  dissemination  and  utility  of  VV&A 
information  of  interest  to  the  DoD  M&S  community. 

5.1  VV&A  Information  on  the  GIG 

The  GIG  is  the  globally  interconnected,  end-to-end 
set  of  information  capabilities,  associated 
processes,  and  personnel  for  collecting, 
processing,  storing,  disseminating,  and  managing 


1  “Validate”  here  is  used  in  the  XML  sense  of  ensuring 
that  the  structure  and  content  of  an  XML  instance 
document  conforms  to  the  specifications  given  in  the 
associated  XML  schema  document(s). 


information  on  demand  to  warfighters,  defense 
policymakers,  and  support  personnel  [19]. 

In  the  GIG,  information  must  be  discoverable  and 
accessible  across  the  enterprise,  dismantling  traditional 
stovepipes  that  have  restricted  information  exchange  in 
the  past.  The  DoD  Net-Centric  Data  Strategy  describes 
the  vision  for  this  net-centric  environment  and  the  data 
goals  for  achieving  that  vision  through: 

—  ensuring  data  are  visible,  available,  and  usable 
when  needed  and  where  needed  to  accelerate 
decision-making; 

—  tagging  all  data  (intelligence,  non-intelligence, 
raw,  and  processed)  with  metadata  to  enable 
discovery  of  data  by  users; 

—  posting  of  all  data  to  shared  spaces  to  provide 
access  to  all  users  except  when  limited  by  security, 
policy  or  regulations;  and 

—  advancing  the  Department  from  defining 
interoperability  through  point-to-point  interfaces  to 
enabling  the  “many-to-many”  exchanges  typical  of 
a  net-centric  data  environment. 

The  GIG  provides  enterprise  services  that  enable  data 
tagging,  sharing,  searching,  and  retrieving  in  support  of 
the  data  strategy. 

The  Net-Centric  Data  Strategy  also  introduces 
management  of  data  within  communities  of  interest 
(COIs)  or  “collaborative  groups  of  users  who  must 
exchange  information  in  pursuit  of  their  shared  goals, 
interests,  missions,  or  business  processes  and  who 
therefore  must  have  shared  vocabulary  for  the 
information  they  exchange”  [19]. 

COIs  address  organization  and  maintenance  of  data 
within  their  domains,  while  tagging  the  data  in  ways 
that  make  the  data  available  for  use  within  the  COI  and 
across  COIs.  COI-specific  metadata  structures  provide 
an  extended  level  of  data  definitions  and  structures.  A 
community  ontology  will  provide  the  data 
categorization,  thesaurus,  key  words,  and  taxonomy. 
The  COI-specific  metadata  structures  and  the 
community  ontology  will  serve  to  increase  semantic 
understanding  and  interoperability  of  the  community 
data. 

The  goal  of  posting  data  to  shared  spaces  uses  metadata 
registries  and  metadata  catalogs.  A  metadata  registry 
contains  information  describing  structure,  format,  and 
definitions  of  data.  DoD  has  established  the  DoD 
Metadata  Registry,  containing  document  formats, 
interface  definitions,  exchange  models  used  by 
systems,  messaging  formats,  symbology,  ontologies, 
and  transformation  services.  The  registry  currently 
incorporates  the  DoD  XML  Registry,  the  Defense  Data 


Dictionary  System,  and  commonly  used  data  reference 
sets.  A  metadata  catalog  contains  instances  of  metadata 
associated  with  individual  data  assets.  In  XML 
parlance,  the  metadata  registry  contains  XML  schema 
files  and  the  metadata  catalog  contains  XML  instance 
documents  conforming  to  the  respective  schema  files. 

To  further  promote  data  discovery,  DoD  created  the 
DDMS.  The  DDMS  “defines  metadata  elements  for 
resources  posted  to  community  and  organizational 
shared  spaces...  The  DDMS  specifies  a  set  of 
information  fields  that  are  to  be  used  to  describe  any 
data  or  service  asset  that  is  made  known  to  the 
enterprise,  and  it  serves  as  a  reference  for  developers, 
architects,  and  engineers  by  laying  a  foundation  for 
Discovery  Services”  [20].  Figure  6  provides  an 
overview  of  the  categories  of  metadata  specified  in  the 
DDMS. 
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Figure  6.  DoD  Discovery  Metadata  Specification 
Overview  (from  [20]) 

This  project  will  support  the  goals  of  the  Net-Centric 
Data  Strategy  with  respect  to  data  relating  to  VV&A  of 
M&S  resources.  Additional  discussion  of  how  the 
project  will  address  the  goals  of  data  visibility, 
discovery,  and  understandability  is  provided  in  the 
following. 

5.2  Visibility,  Discovery,  Understandability 

To  promote  discovery,  this  project  will  leverage  the 
DDMS  XML  vocabulary  through  reference  to  the 
DDMS  namespace.  Another  effort  is  in  progress  to 
identify  DDMS  metadata  applicable  to  DoD  M&S 
products  and  resources  and  to  identify  additional 
metadata  needed  for  M&S  purposes  that  will 
supplement  DDMS  requirements  (and  may  be 
proposed  as  extensions  to  DDMS).  Also,  that  work  will 
be  leveraged  to  identify  DDMS  and  M&S  metadata 
that  can  be  reused  in  the  DVDT  XML  schema  and  in 
the  document  instances  conforming  to  that  schema. 
Finally,  the  project  will  also  determine  if  certain 
metadata  specific  to  VV&A  documentation  should  be 


proposed  as  extensions  to  the  DDMS  for  general  use 
across  the  enterprise. 

To  further  promote  the  Net-Centric  Data  Strategy  goal 
of  enabling  the  data  to  be  understood,  this  project  will 
explore  application  of  other  web-based  standards  for 
describing  information.  Tim  Berners-Lee’s  vision  for 
the  World  Wide  Web  is  creation  of  a  web  of 
knowledge,  termed  the  Semantic  Web  [21].  This  is 
being  addressed  through  research  and  development  of 
standards  that  provide  representation  of  stronger 
semantics  in  web-based  information,  in  line  with  the 
Net-Centric  Data  Strategy  goal  of  enabling  data  to  be 
understandable  by  users  and  applications,  both 
structurally  and  semantically.  Current  and  emerging 
Semantic  Web  standards  include  the  Resource 
Description  Framework  (RDF),  RDF  Schema,  Web 
Ontology  Language  (OWL),  and  Semantic  Web  Rule 
Language  (SWRL),  among  others. 

5.3  Strong  Semantics 

For  future  growth  in  automation  of  VV&A  processes 
and  information,  the  project  will  develop  a  formal 
ontology  that  can  be  employed  by  software  and 
humans  attempting  to  discover  M&S 
components/resources  and  to  assess  suitability  of  those 
components/resources  for  the  desired  purpose.  We  will 
investigate  the  current  state  of  defined  VV&A 
taxonomies,  processes,  and  artifacts  to  design  an  initial 
VV&A  ontology  describing  important  concepts, 
properties,  relationships,  constraints,  and  business 
rules.  The  ontology  will  be  developed  using  Web 
Ontology  Language  (OWL)  and  other  Semantic  Web 
standards  as  deemed  appropriate. 

The  VV&A  ontology  work  will  establish  a  technical 
case  for  application  of  formalized  semantics  relating  to 
VV&A  processes  and  records.  The  formalisms  will 
include  the  above  metadata  (XML  schema)  describing 
M&S  data  and  products;  but  will  extend  the  data 
modeling  to  provide  deeper  description  of  the  concepts. 
The  work  will  review  prior  VV&A  research  and 
development  to  develop  an  initial  taxonomy  of  M&S 
artifacts  and  VV&A  processes  and  artifacts  (for 
example,  see  [22]  and  [23]).  The  taxonomy  will  then  be 
extended  to  include  properties  that  reflect 
interrelationships  across  classes  or  categories  of 
concepts.  For  example,  a  taxonomy  of  M&S  systems 
may  classify  systems  by  use  as  training  or  analysis 
systems,  with  possibly  a  third  classification  for  systems 
that  can  be  used  for  both  purposes.  However,  an  M&S 
system  accredited  for  training  may  also  be  accredited 
for  analysis  purposes,  but  only  under  certain 
constraints  and  conditions  in  its  employment. 


The  project  will  design  and  develop  an  ontology 
expressing  VV&A  information  established  in  VV&A 
processes  and  M&S  documentation.  Previous  markup 
language  work  and  the  Defense  Standard  Practice  [2] 
provide  an  excellent  starting  point  for  defining  classes 
and  properties  in  the  ontology.  Concentration  will  be 
on  classification  schemes  that  will  support  semantic 
discovery  of  VV&A  metadata  describing  M&S 
resources  as  well  as  providing  an  ability  to  perform 
logical  inferences  on  the  information  to  relate  VV&A 
information  to  user  or  system  requirements  in  obtaining 
and  using  needed  M&S  resources.  In  addition  to  the 
ontology  itself,  the  work  will  produce  a  technical  report 
on  the  ontology  design,  including  design  decisions  and 
trade-offs  made  during  the  effort. 

More  expressive  ontologies  can  describe  not  only  the 
classes  and  their  properties  relevant  to  VV&A 
information,  but  relationships  across  classes  that  cannot 
be  represented  in  a  strict  hierarchical  structure  as 
defined  by  XML  Schema.  In  the  literature  on  ontology 
design,  the  importance  of  determining  the  domain  and 
scope  of  the  ontology,  to  include  identification  of 
questions  that  a  knowledge  base  built  from  the 
ontology  should  be  able  to  answer,  is  emphasized  [24]. 
This  level  of  semantic  sophistication  will  be  needed  to 
enable  humans  and  software  to  better  access  and 
employ  VV&A  information  about  M&S  resources  as 
the  community  moves  toward  GIG  service-oriented 
and  net-centric  architectures. 

6.  Summary 

This  paper  updated  the  Systems  Engineering 
Community  about  the  DoD  M&S  Project  titled, 
’’Standardized  Documentation  for  VV&A’’.  It  provided 
information  on  the  development  of  standardized 
content  and  format  requirements  for  four  core  VV&A 
documents,  the  technology  development  efforts  to 
automate  those  templates  to  ensure  standardization 
across  the  DoD  and  Military  Departments,  and  the 
work  underway  to  identify  VV&A  metadata  that  will 
enable  the  sharing  of  information  across  all  M&S 
Communities  via  GIG  service-oriented  and  net-centric 
architectures.  Additionally,  gaps,  objectives,  and 
actions  from  the  Common  and  Cross-Cutting  Business 
Plan  and  the  DoD  Acquisition  M&S  Master  Plan  were 
identified.  Finally,  a  project  management  overview 
including  a  concept  of  operations  was  provided. 

If  you  would  like  to  become  a  DVDT  beta  tester  or  use 
the  DVDT  to  support  a  VV&A  project,  you  can  contact 
any  one  of  the  authors  to  obtain  more  information. 
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•  IRM  Process  Background 

•  Integrated  Risk  Assessment  Overview 

•  IRM  Improvement  Efforts 

•  Linkage  with  Air  Force  level  efforts 

•  Status  of  Risk  Management  Actions 
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IRM  Process  Background 


*m  pcs, 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  IRM  survey  revealed  confusion  in  Acquisition 
Weapon  System  (WS)  Program  Offices 

-  Poor  results 

-  Lack  of  understanding 


•  Inconsistent  process  across  WS  life  cycle 

-  Varying  levels  of  rigor 

-  Process  and  knowledge  base  not  documented 
across  phases 


3 


IRM  Process  Background 


*m  pcs, 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Pockets  of  excellent  work 


-  ACE  pre-contract  award  risk  workshops 

-  Engineering  (EN)-facilitated  Integrated  Risk 
Assessments  (IRAs) 

-  Finance  (FM)-executed  schedule  and  cost  risk 
assessments  for  annual  Program  Office  Estimate 
(POE) 


•  Efforts  not  well  coordinated  and  IRA  execution 
was  not  prioritized 


IRM  joint  process  between  Engineering  and 
Finance.  Program  management  not  involved. 
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U~£  Hilt  FOm&E 


Integrated  Risk  Management 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


Risk  Planning 


Risk  Assessment 


IRA:  Cornerstone  of 
IRM  at  ASC 


Risk 

Handling 

- / 

r - \ 

Risk 

Monitoring 

V _ J 


Risk  Documentation 
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What  is  an 


Integrated  Risk  Assessment? 


pcs, 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Identifies  and  analyzes  program  risks  against 
performance,  schedule  and  cost  objectives 

-  Reveals  impacted  resources 

*  Performance  -  Schedule  -  Cost 

-  Develops  more  realistic  schedule  and  cost  estimates 
via  Monte  Carlo  simulations  with  revealing  90% 
confidence  interval 

-  Should  coincide  with  the  annual  life  cycle  Program 
Office  Estimate  (POE) 

•  Two  Segments 

-  Risk  identification  (qualification  -  ASC/AE  and  FM) 

-  Evaluation  and  analysis  (quantification  -  ASC/FM) 
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Risk  Assessments 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


\ 


Pre-award,  pre-MS  B 

Input  Scope,  Purpose,  Consequence  Definitions _ ^ ^workshop  (ASC/AE)^ 


i  r 


Evaluation 


a 


Analyses 


iL 


Identification 


F+4-  —  ■ 

■ 

IMS  \ 

Technical  \ 

;;  r_ 

w 

Performance  1 

m 

Assessment : 

r 

Assessment/ 

P/CS^ 


Risks: 

•Technical 

•Schedule 

•Cost 


POE 

Cost 

Estimate 


Risk  Impacts: 

•Technical 

•Schedule 

•Cost 

’•  Best  case 

•  Worst  case 

•  Most  likely 


Risk+®  Impact^*  Crystal  Ball 


Output 
Risk  Matrix 
"Schedule 
Cost  Estimate 
List  of  Risks 
Risk  Report 
Presentations 


Integrated  Risk  Assessment  (AE,  EN  and  FM)^' 


Outputs  from  both  venues  lead  to  more  effective  risk  handling  and  monitoring 


*  Government  Owned.  ®  CS  Solutions.  ProjectGear.  Inc.  and  ®  Decisioneering 


IRM  Process  Background,  cont 

™  Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today  “ 


•  ASC  Commander  Policy 

“Complete  an  annual  IRA,  ideally  in  conjunction  with 
the  annual  program  life  cycle  cost  estimate  (POE)  to 
ensure  risks/risk  handling  are  quantified  and 
appropriately  budgeted.” 

•  Policy  is  not  followed 

-  Insufficient  manpower  to  execute  policy 

•  Both  in  wings  and  staff  to  support 

-  No  tracking  of  IRA  activities 

•  Inconsistent  IRA  and  POE  requirements 

-  POE  policy  allows  requirement  waiver  for  programs 
meeting  certain  criteria 

-  No  similar  policy  for  IRAs 


8 


IRM  Improvement 


urn  pcs, 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Risk  Staff:  program  management, 
engineering,  and  financial  management  began 
risk  management  process  advancement 
endeavors 


•  ASC  Leadership  process  improvement  offsite 
(AFS021 /Balanced  Scorecard)  identified  two 
risk  management  initiatives 

•  Consolidated  staff  and  Balanced  Scorecard 
initiatives  into  single  effort 

-  “Improve  risk  vision,  advocacy  and  processes” 


9 


i  J  \ 

AF-Level  Process  Improvement 

Hilt  l~ 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 

•  Develop  and  Sustain  Warfighter  Systems 
(D&SWS)  Process  Improvement  Team 

-  Continuous  Capability  Planning  Sub-Process  Team 

•  Recommend  process  improvements,  initiatives  and  metrics 

-  RM  is  initiative  due  to  high-level  interest 

•  Labeled  as  “enduring  process”  throughout  life  cycle 

•  Objective  to  standardize  processes,  definitions,  tools 

•  SAF  ACE  identified  as  process  owner  and  Implementation 
Team  Lead 
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Risk  Management  Improvement 
Completed  Actions 


V 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Refine  organizational  responsibilities 

-  AE/EN/FM  joint  owners  of  IRM  process 

-  AE  designated  as  ASC  risk  management  lead 

-  EN  IRM  Tech  Expert  moved  to  AE  to  increase 
effectivity,  efficiency,  and  consistency 

•  Ensured  risk  aspects  of  Probability  of 
Program  Success  (PoPs)  assessment  tool 
was  incorporated  into  risk  assessments 

•  Ensure  uniform  risk  training  across  ASC 
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Risk  Management  Improvement 

Current  Action 


v 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Improving  efficiency  and  effectiveness  of  IRA 
process 

-  Align  need  for  high-confidence  programs  with 
manpower  limitations 

-  Prioritize  programs  for  IRAs  beginning  Winter  07 

•  Beta  test  using  FM’s  Apr  06  PoE  waiver 
process  data  to  determine  applicability  to  IRA 
waivers 

-  Updated  data  with  Wing  Commanders  and 
Directors  of  Engineering 

-  Determined  PoE  waiver  good  starting  point  for  IRA 
waiver  or  tailored  IRA 
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POE  Waiver  Criteria 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 

•  ACAT  III  IRA  Waiver  Process 

-  Low  cost/risk 

-  Cost  Contract 

-  Firm  Fixed  Price  <  $50M 

-  Time  and  Materials  contract 

-  Level  of  Effort  programs 

-  Programs  in  last  year  of  effort 

-  Short  duration  programs  (1  year  or  less) 
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Risk  Management  Improvement  v 
***  Current  Actions,  cont 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 

•  Complete  waiver  process  of  ACAT  III 
programs 

•  Prioritize  remaining  programs  (ACAT 
I,  II,  and  required  ACAT  III) 

-  Event  driven  rather  than  annual 

-  Emphasize  high  risk  and  new  programs 

•  Develop  schedule  for  IRA  execution 

•  Update  ASC/CC  IRM  policy 

IRA  waiver  does  not  eliminate  need  for  day-to-day  risk  management 


14 


IRA  Prioritization  Schedule 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 

•  Emphasis  new  programs 

•  Ensure  IRA  completed  early  in  program 

-  Prior  to  PDR 

-  Complete  early  assessment  of  performance, 
schedule  and  cost  risks  and  impacts.  And  Maintain! 

-  Establish  robust  risk  management  practices  at  the 
onset  of  the  program 
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Risk  Management  Improvement 
Ongoing  Actions 


v 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Improve  and  document  total  program  Life 
Cycle  risk  process 

•  Ensure  consistency  with  DoD,  DAU,  AFIT, 
CSE,  and  INCOSE 

•  Create  ASC-wide  risk  IPT  with  reps  from 
AE/EN/FM  and  each  WS  program  office 

•  Increase  knowledge  and  awareness  of  risk 
management 

•  Develop  cadre  of  trained  facilitators 

•  Evaluate  adequacy  of  available  manpower 
resources 
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Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


Comments/Questions?' 
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Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


Backup 
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IRM  Survey  Results 


urn  pcs, 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  IRM  survey  revealed  confusion  on 
requirements  in  Wings 

-  Poor  results 

-  Lack  of  understanding 

•  Inadequate  manpower  in  Wings  and  staff 
organizations  to  support  IRA  policy 

•  Disconnects  between  annual  Program  Office 
Estimate  (POE)  and  IRA  policy  letters 

-  IRA  and  POE  requirements  are  linked 

*  Results  of  IRA  input  to  POE 

*  Can  complete  IRA  without  POE;  however,  POE  is  not 
considered  complete  without  IRA 

*  Identification  of  risks  alone  does  not  constitute  an  IRA 

-  Current  policy  letters  are  out  of  sync 

*  POE  policy  allows  waiver  under  certain  conditions 

*  IRM  policy  allows  for  no  IRA  waiver 


Risk  Workshop  vs.  IRA 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Significant  difference  in  AE  Risk  pre¬ 
award  workshops  and  Facilitated  IRA 

•  1  day  vs  2  weeks 

•  Focus  primarily  on  high  level  programmatic  risk 
assessment  vs  total  risk  assessment  (cost, 
schedule,  performance) 

-  Shallow  dive  vs  deep  dive 

-  Risk  of  getting  on  contract  vs  total  program 
risks  (cost,  schedule,  performance) 
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Technical  Performance 
Assessment 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Insert  description 


22 


Schedule  Risk  Analysis  (SRA) 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 

•  Risks  identified  during  IRA  are  quantified  in  and 
added  to  the  Integrated  Master  Schedule  (time) 

•  Accomplished  for  critical  path  elements  (time 
constraints  may  preclude  expanding  SRA  to 
other  elements) 

•  Best  case,  most  likely  and  worst  case  results 
input  to  Risk+  schedule  assessment  tool,  Monte 
Carlo  analysis  run 

•  Results  in  additional  time  (hrs,  months,  etc.) 
and  dollars  needed  for  higher  confidence 
schedule 
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Cost  Risk  Analysis  (CRA) 

™  Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Risks  identified  during  IRA  are  quantified  in  and 
added  to  the  cost  estimate  (dollars) 

•  Accomplished  at  lowest  WBS  element 
appropriate 

•  Best  case,  most  likely  and  worst  case  results 
input  to  Crystal  Ball  cost  assessment  tool, 
Monte  Carlo  analysis  run 

•  Results  in  additional  costs  required  for  higher 
confidence  estimate 
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Risk  in  the  Acquisition  Life  Cycle 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 

I 


Za\ Zcl 


Concept 

Technology  | 

System  Development 

Production  & 

Operations  & 

Refinement 

&  Demonstration 

Deployment 

Concept  Decision 

Development. 

Design  Readiness 
^  Review 

FRP  Decision 
X/  Review 

Support 

ASC/AE-facilitated  Risk  Workshop* * 


ASC/EN-  and  FM-facilitated  Integrated  Risk  Assessment 


Pre-award,  Pre-MS  B 

Output:  Risks  identified  and  plotted  on 

5x5  matrix 

Duration:  1-3  days 

Location:  ASC/AE  facility 

Involves  Program  Office,  user, 

contractor  (if  sole  source) 

Technical  Analysis  accomplished  on 
ability  to  execute,  schedule  analysis 
accomplished  on  getting  on  contract, 
not  program’s  IMS 
ASC/CC  Policy:  N/A 


- > 

Post-award,  annual  requirement  throughout  acquisition  life  cycle 
Output:  Risks  identified  and  plotted  on  5x5  matrix,  risk  impacts  quantified, 
risk  impacts  analyzed  via  statistical  analysis,  additional  time  and  budget 
required  to  complete  program  calculated  and  added  to  IMS  and  POE 
Duration:  2  weeks  not  including  prep  time  and  post-workshop  analysis  time 
Location:  Offsite 

Involves  Program  Office  (all  functionals),  contractors,  subs,  users,  subject 
matter  experts,  advisors 

Technical,  Schedule  and  cost  analyses  accomplished  against  ability  to  meet 
contract  requirements 
ASC/CC  Policy:  05-004 


*Can  occur  at  any  time  throughout  the  | 
life  cycle  in  conjunction  with  contractual  ■ 
actions  (pre  MS  B,  LRIP,  pre  MS  C,  etc)1 

Day-to-day  risk  management  should  occur  jointly  with  the  contractor 


Recent  Policy  Directives 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  ASC/CC  Policy  Memo  05-014:  PEO  Policy  for 
Systems  Engineering 

-  Requires  more  rigorous  systems  engineering  with 
risk  management  as  a  key  aspect 

-  https://www.asc.wpafb.af.mil/policy_letters/policym 
emo05-014.pdf 

•  ASC/CC  Policy  Memo  05-003:  Policy  on  Life  Cycle 
Estimates 

-  Programs  to  provide  Life  Cycle  Cost  Estimates 
including  integrated  risk  assessments  reflecting 
90%  confidence  of  meeting  our  commitments 

-  https://www.asc.wpafb.af.mil/policy_letters/policym 
emo05-003.pdf 


Recent  Policy  Directives 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  ASC/CC  Policy  Memo  05-004,  Policy  on  Integrated 
Risk  Management 

-  Requires  annual  Integrated  Risk  Assessment 


•  ASC/CC  Policy  Memo  06-007,  Policy  on 

Environment,  Safety  and  Occupational  Health 
(ESOH)  Programmatic  Risk  Management 
Integration  into  Acquisition  and  Systems 
Engineering  Processes 

-  Requires  annual  programmatic  ESOH  risk 
assessments  in  conjunction  with  the  IRA 

-  https://www.asc.wpafb.af.mil/policy_letters/policym 

emo06-007.pdf  n 


Probability  ►  Higher 
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Sample  Risk  Definitions 

Dominant  Air  Power :  Design  For  Tomorrow... Deliver  Todav 

Tailorable  Risk  Grid 


Consequence 


Negligible  (N) 


Higher 


91-100% 


61-90% 


3 

41-60% 


11-40% 


0-10% 


Minor  (M)i) 


Cost  =  <  1%  Variance 

Schedule  =  <  1  WkVar 

Technical  =  Meets 
Performance 


Cost  =  <  1%  Variance 

Schedule  =  <  1  WkVar 

Technical  =  Meets 
Performance 


Cost  =  <  1%  Variance 

Schedule  =  <  1  WkVar 

Technical  =  Meets 
Performance 

Cost  =  <  1%  Variance 

Schedule  =  <  1  WkVar 

Technical  =  Meets 
Performance 


Cost  =  <  1%  Variance 

Schedule  =  <  1  WkVar 

Technical  =  Meets 
Performance 


Cost  =  1-5%  Variance 

Schedule  =  1-4  WkVar 

Technical  =  Minimal 
Impact  to  Performance 


Cost  =  1-5%  Variance 

Schedule  =  1-4  WkVar 

Technical  =  Minimal 
Impact  to  Performance 


Cost  =  6-10%  Variance 
Schedule  =  5-8  WkVar 
Techi/Performance  = 
Acceptable  Work¬ 
around 


Cost  =  1-5%  Variance 

Schedule  =  1-4  WkVar 

Technical  =  Minimal 
Impact  to  Performance 

Cost  =  1-5%  Variance 

Schedule  =  1-4  WkVar 

Technical  =  Minimal 
Impact  to  Performance 


Cost  =  6-10%  Variance 
Schedule  =  5-8  WkVar 
Tech/Performance  = 
Acceptable  Work¬ 
around 

Cost  =  6-10%  Variance 
Schedule  =  5-8  WkVar 
Tech/Performance  = 
Acceptable  Work¬ 
around 


Cost  =  1-5%  Variance 

Schedule  =  1-4  WkVar 

Technical  =  Minimal 
Impact  to  Performance 


Cost  =  6-10%  Variance 
Schedule  =  5-8  WkVar 
Tech/Performance  = 
Acceptable  Work¬ 
around 


Cost  =  11-20%  Variance 
Schedule  =  9-12  WkVar 
Tech/Performance  = 
Degraded 

Cost=  11-20%  Variance 
Schedule  =  9-12  WkVar 
Tech/Performance  = 
Degraded 


Cost  =  11-20%  Variance 
Schedule  =  9-12  WkVar 
Tech/Performance  = 
Degraded 


Cost  =  >  20%  Variance 
Schedule  =  >  12  Wk 
Variance 

Tech/Performance  = 
Impacted 


Cost  =  >  20%  Variance 
Schedule  =  >  12  Wk 
Variance 

Tech/Performance  = 
Impacted 


>8 


Legend:  Low  E 


□  Medium  E 


High 
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Integrated  Risk  Management  Survey 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


*  Does  your  program  have  a  Risk  Management  Plan 
(RMP)? 


•  Does  your  program  have  an  Integrated  Master 
Schedule  (IMS)? 


•  Does  your  program  have  a  current  Integrated  Risk 
Assessment  (IRA)? 


29 


Survey  Top  level  Findings 

i  Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today  > 

*  Lack  of  common  understanding  of  requirements,  policy 
and  utility  of  these  items 

-  Many  have  “never  heard  of  the  IRA  process” 

-  Many  believe  “I’ve  got  Risk  covered” 

•  I  did  an  AE  Risk  Workshop 

•  I’ve  got  my  risks  plotted  on  a  matrix! 

•  Yes  I  had  an  IRA.  It’s  scheduled  next  year! 

*  Organizations  lack  training  in  RMPs,  IMSs,  and  IRAs 

*  Programs  are  not  following  current  ASC  IRM  policy 
relating  to  RMP  (64%),  IMS  (67%),  IRA  (56%) 

-  Confusion  on  how  to  answer  questions 

-  RMP  and  IMS  stats  seem  to  be  inaccurate,  IRA  stat  is  inaccurate 


Program  Risk  Not  Adequately  Understood  or  Managed  at  ASC 
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D&SWS  Integrated  “To  Be” 

™  Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today  ™ 


Governing  Sub-Processes 


Enabling  Sub-Processes 


Life  Cycle  Management  of  Data 


Attributes 


Minimize  Total 
Ownership  Costs 

Capability  Focus 
Throughout  Life  Cycle 

High  Confidence 
Programs 

Transition  Mature 
Technology 

Tight  Stakeholder 
Partnerships 

Right  Data  for  Right 
Decision  at  Right  Time 
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Enduring  Processes 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


Identification  &  Management  of  Requirements 


Engineering  Analysis  and  Modeling 


Architecture  Development  and  Interoperability 


Supply  Chain  Management 


Test  and  Evaluation 


Contracting 


Cost  Estimating 


Risk  Management 


Integration  and  Systems  Engineering 


Life  Cycle  Management  of  Data 
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Air  Force  Modeling  &  Simulation 

Training  Toolkit 
(AFMSTT) 


Advanced  Net  Centric 
Simulation  for 
Aerospace  Command  & 


U.S.  AIR  FORCE 


Control 

Ms.  Kim  Kendall 
AFMSTT  Program  Manager 
753d  Electronic  Systems  Group 
Electronic  Systems  Center 
Hanscom  AFB,  MA 
24  October,  2007 


Overview 


U.S.  AIR  FORCE 


•  Winning  the  GWOT  -  M&S  a  Force  Multiplier 

•  Why  Netcentric  Operations? 

•  Moving  to  the  Global  Information  Grid 

•  Building  a  Foundation  for  Netcentric  Operations 

•  Systems  Engineering  “Best  Practice”  Approach 


Integrity  -  Service  -  Excellence 


U.S.AIR  FORCE 


AFMSTT  Supported 
Disciplines 


TRAINING 


MSN  REHEARSAL 


ACQUISITION 


EXPERIMENTS 


WARGAMING 


FUTURE 


ANALYSIS 


PLANNING 


TESTING 


Integrity  -  Service  -  Excellence 
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Win  the  GWOT 
AFMSTT  a  Force  Multiplier 


•  Realistic/Consistent  Training 

•  Integrated  Live,  Virtual,  Constructive 

•  Dynamic,  interactive  planning  &  decision  support 

•  M&S  transparent  to  training  audience 

•  Mission  Rehearsal 

•  On  demand,  on  location  training 

•  Flexible  scenario  generation 

•  Risk  Reduction 

•  Safer  environment  for  Warfighter 

•  Interact  with  real  world  C2  systems 


AFMSTT  supported  14  major  and  26  associated  test/integration  events  last  year! 


Integrity  -  Service  -  Excellence 
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Constructive  Air  Power  Simulation 

Supporting 

Joint/Service  Battlestaff 


Exercise  Support 

•  Ulchi  Focus  Lens 

•  BLUE  FLAG 

•  Austere  Challenge 

•  Joint  RED  FLAG 

•  Terminal  Fury 

•  Ardent  Sentry 


Experimentation  Support 

•  Joint  Expeditionary  Force  Experiment  (JEFX) 

*  Coalition  Warrior  Interoperability  Demo  (CWID) 


AFMSTT 


Mission  Rehearsals 

•  Unified  Endeavor 

*  Global  Thunder 


Acquisition  Support 

TBMCS  Testing 
•  AF-ICE 


Integrity  -  Service  -  Excellence 


Why  Net-Centric 
Operations? 


U.S.AIR  FORCE 


•  Net-Centric  Operations  is  the  ability  to: 

•  Rapidly  collect  and  share  appropriate  data  in  a  collaborative 
environment 

•  Recognize  significant  data 

•  Understand  the  data 

•  Efficiently  make  better-informed  decisions  by  yourself  or  in  a 
collaborative  environment 

•  Rapidly  act  (or  not  act)  on  decisions 

•  Rapidly  get  feedback 

•  Understand  services  available 

•  Efficiently  use  those  services  (or  capabilities) 

•  Efficiently  provide  services  for  others  in  a  manner  consistent 
with  your  mission 

J  Derived  from:  Network  Centric  Warfare,  Alberts,  Garstka,  Stein 


Integrity  -  Service  -  Excellence 
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AFMSTT  with  the  GIG 


Integrated  M&S  and  C2  Environments  that  support 

Net  Centric  Operations  (NCO) 


Integrity  -  Service  -  Excellence 
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Moving  to  Netcentricity 
using  NESI  tenants 


Net-centric  Enterprise  Solutions  for  Interoperability  (NESI) 

•  Implementation  guidance  to  facilitate  the  design, 
development  and  usage  of  information  systems 
for  net-centric  warfare 

•  Effective  for  migrating  deployed  applications 
using  a  phased  approach 

•  Based  on  industry  best  practices 

•  Cross-Service  effort  between  Air  Force  (ESC) 
and  Navy  (PEO  C4I  &  Space) 

•  Army  &  DISA  participated  informally 


Integrity  -  Service  -  Excellence 
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Systems  Engineering 
“Best  Practices”  Approach 


•  Incremental  Approach 

•  No  wholesale  re-write  of  code  base 

•  No  impact  to  current  operations  tempo/event 
support 


*  Collateral  requirements 

•  No  impact  to  toolkit  performance 

•  No  increase  in  AFMSTT  footprint 


Integrity  -  Service  -  Excellence 


Systems  Engineering 
Best  Practices”  Approach 
(Cont) 


U.S.  AIR  FORCE 


•  Architecture  requirements 

•  Integrate  within  Current  Acquisition  Framework 

•  Design  for  Location  Independence 

•  Increase  Collaboration 

•  Separate  business,  data  and  presentation  logic 

•  Service  Oriented  Architecture  (SOA) 


Integrity  -  Service  -  Excellence 
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Agile  Acquisition 
Development  Framework 


© 

Simulation  Event, 

Exercise 

or 

Test  Activity 


AFAMS  places 
CRs  and  DRs  into  DB 


Simulation  Centers  and 
Numbered  AF  Generate 
CRs  and  DRs  f“ 

I'X 


/ 


/ 


/ 

/  / 
/ 

/ 

/ 


\ 


\ 


-  CR/DR  Clarifications'  \  ^ 

-  Sprint  Review  Invites  \  i 


K  1  ESC  Oversight 

/Z4 

/  /  -  CR/DR  Clarifications 
N/  /  -  Sprint  Review  Invites 

\ ^  -  DR  submissions 

^  -  Estimates 


\7 


\ 

\\ 

\ 

\ 
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Warfighter  Advocate 
Assists  with 
CR&DR 
communication 


-  CR/DR  Clarifications 

-  Assessments  (S,M,L) 

-  Need -by  Event/ Date 


-A* 

Developer 

CCB 


y 


-  CR/DR  Clarifications 
Developer  Questions 


ijfr  :  V 


Exercise  Support  IPT 
Adds  CR  and  DR 
Context 


Development  & 
Maintenance  IPT 
Updates  Code 


Tech  Leads 
Break  Down 
Stories 
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Exposing  Data 


U.S.  AIR  FORCE 


SIMULATION: 

Shared  Memory 


SERVICE 


-  -OGSIM 

-  MTK 
-ASTAB  Display 

-  Order  Entry  (OE) 
-New  Capabilities 


Integrity  -  Service  -  Excellence 


^*fC3  building  a  Foundation  for 

Netcentric  Operations 
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Composable 

Collaborative 

Tailorable 


Data 

Services 

Layer 


Dash  Board 


Reporting 


M2M 


-  I  £•  il 
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Publish 


Publish 


Publish 


r 

o 

o  o 

O 

Logistics 

^  Order  Stack  C2 

O 

SA 

V 

Parametric  Enumerations 

Missions 

J 

Existing  Data 


Other  Models 
^GSI^AC^S^ 


Flat  Files 


User  Input 


RDBMS 
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Real  Time 
Information 


Game  State 
Data 


Siloed 

& 

Complex 


Benefits/ 
New  Possibilities 


U.S.  AIR  FORCE 


An  AFMSTT  data  service  will  promote... 

•  Access  to  virtual  battlespace  information  using 
established  commercial  standards 

•  SOAP/WSDL 

•  REST 

•  XML  message  protocol 

•  Improved  Model  Controller  Efficiency 

•  Provides  Integration  Engine  for  LVC  environment 

•  Enhanced  capability  for  analysis/agile  acquisition 


Integrity  -  Service  -  Excellence 


Benefits/ 
New  Possibilities 


U.S.  AIR  FORCE 


and  unlock  untapped  investment! 


•  Easy  integration  of  new  capabilities  as  training 
missions  change 

•  Flexible,  Adaptive  Functionality  Reuse 

•  Location  Independence 

•  Reduced  set-up  time  &  travel  costs 

•  Collaborative  Event  Planning/Managing 

•  Reachback  to  center  of  excellence 


Integrity  -  Service  -  Excellence 
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New/Emerging 
Netcentric  Capabilities 


•  Enhanced  Reachback 

•  24X7  Help  Desk 

•  FAQ  Portal 

•  Re-architected  Distributed  Mission  Planning 
Workstation 

•  Java  Message  Service  replaced  direct  queries 

•  Preserved  core  business  logic 

•  Solution  was  transparent  to  end  users 

•  JTEN  node  connection  from  ESC  integration 
facility 


Integrity  -  Service  -  Excellence 


Providing  Capability  to  the  Warfighter 


i  We’re  Proud 


We’re  Pumped 


U.S.  AIR  FORCE 


Summary 


•  M&S  provides  a  low  cost  option  for  validating  war 
fighter  missions  pre-execution,  real  world 
systems,  and  a  broad  array  of  AF  disciplines 
(acquisition,  test,  etc) 

•  Adaptation  to  changing  GWOT  mission  is  critical 

•  Modernization  can  be  achieved  through  sensible 
“localized”  wins  focused  on  scaling  in  a  net-centric  way 

•  Provides  continued  innovation  in  netcentric 
capabilities  to  provide  relevant  capability  to  the 
user  anytime,  anywhere 


Bringing  a  very  valuable  legacy  simulation  into 

the  21st  Century !! 


Integrity  -  Service  -  Excellence 
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Questions? 
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Airborne  Signals  Intelligence  Payload  (ASIP) 
Integrated  Risk  Management 


Presented  to: 

10th  Annual  Systems  Engineering  Conference 

24  October  2007 


Legend 
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Platform 


scan 


Gil  SMU 
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Dan  Bolek 
659  AESS/TX 
937-255-9973 

daniel.bolek@wpafb.af.mil 
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•  ASIP  Program 

•  ASIP  System  of  Systems 

•  Risk  Terminology 

•  ASC/EN  SE  Tool  Set 

•  Integrated  Risk  Management 

•  Integrated  Risk  Assessment  (IRA) 

•  ASIP  IRA  Risk  Summary 

•  ASIP  SoS  Risk  Management 

•  ASIP  Risk  Management  Model 

•  Best  Practices 

•  ASIP  Risk  Management  Results 

•  Risk  Metrics 

•  Take  Aways 


Airborne  Signals  Intelligence  Payload 

(ASIP)  Program 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  USAF  ACAT  II  SDD  program  under  OSD  T&E 
oversight 

-  Currently  in  developmental  flight  test  on  U-2 


Payload  collects  &  processes  COMINT, 
ELINT  &  special  signals  targets 

Managed  as  a  system-of-systems  (SoS) 


-  Integrated  master  schedule 

-  Interface  control 


Risk 

Management 


-  RISK  MANAGEMENT 


I  -  I  — T 


Risk  Risk  Risk 

Planning  Assessment  Handling 


Risk 

Monitoring 


Risk 

Identification 


Risk 

Analysis 


ASIP  System  of  Systems* 
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NG  -  ESL 
(Sensor) 
659  AESS 


L-3  COM 


(Data  Link) 
674  AESF / 


578  ACSS 


LM-A 

(U-2) 

674  AESF 


* 


Managed  at  WPAFB 
Managed  at  WRALC 


“...A  system  of  systems  is  a  set  or  arrangement  of  interdependent  systems  that  are  related  or 
connected  to  provide  a  given  capability.  The  loss  of  any  part  of  the  system  will  significantly 
degrade  the  performance  or  capabilities  of  the  whole....”  (DAG  1.2.1) 


Terminology 
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•  Risk:  A  negative,  future  event  that  may  occur,  causing 
an  execution  failure  in  the  program  and  you  are  able  to 
estimate  the  probability  and  consequence  (between  0- 
100%  probability  of  occurrence) 

•  Issue:  A  negative,  future  event  that  is  certain  to  occur 
and  will  have  a  negative  impact  on  the  program  (100% 
probability  of  occurrence) 

•  Problem:  A  negative  event  that  has  already  occurred  (a 
risk  that  has  come  to  fruition) 

•  Concern:  A  negative,  future  event  that  may  occur,  but 
you  have  insufficient  information  to  quantify  the 
probability  and  consequence 
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ASC/EN  SE  Tool  Set 
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Integrated 

Risk 

Management 


1 

■ 

Weapon  Sretem  IntsuriEv 

3 '  1 

Expanded 

Airworthiness 

Crirle‘ria 


Mil-Hdbk-1798 

Mtl-fldbk1783 

M«l-4Hdbk-B7244 

MECSIP 

EMSIP 

AVIP 

Mil-Std  for 

Mil-Std-15-30 

Mil-Std  for 

Mil-Std  for 

A/W  Criteria 

AS  IP 

MECSIP 

EISISIP 

Mil-Std  for 
AVIP 


Use  SE  Tool  Set 
to  Derive  Program 
Specific 
Applications 


Program  Unique  Products 


■  Acquisition  Strategy 

■  Systems  Engineering  Plan 

•  SE  RFP  Template 

•  Development  Contracts 

-  Statement  of  Work 

-  Specification 

-  IMP/IMS 

•Production  Contract(s) 
•Sustainment  Activities 


"...  SE  tools  and  processes  available  to  assist  program  teams  in  structuring  a 
program  specific  SE  approach 


Integrated  Risk  Management  (IRM) 
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IRM  is  preferred  method  at  ASC 

-  Part  of  acquisition  strategy 

-  Joint  government/industry 


DEPARTMENT  OF  THE  AIR  FORCE 

MCADOUARU  M=  AHlQrML/nCAl.  SYSTEMS  C  NTEfl  AFVC) 


[4  Uoo  2 oof 


MEMORANDUM  FOR  ASC  WING  COMMANDHRS/DIRECTORS 

ASC'DIRTCT  REPORT  GROJF  COMMANDERS 
ASC  SENIOR  FUNCTIONALS 

FROM:  ASC/CC 

STJnJF.CT.  Pulley  ou  Integrated  Risk  Management  (ASC/CC  Policy  Memo  ^05  00-1) 


-  Cross-functional 

-  Single  program  risk  deck 

-  Documented  Risk  management  plan  « — { 

-  Integrated  Risk  Assessment  (IRA) 
accomplished  in  conjunction  with 
annual  life  cycle  cost  estimate 


: .  A  key  element  ofmanagine  any  complex  program  is  the  management  of  risk.  C  urrent  Air 
Fotec  guidance  states  program  managers  should  conduct  a  comprehensive  risk  analysis  3nd  shall 
prepare  and  maintain  a  rnrrcnl  R:sk  Management  Plan.  The  Aircraft  Program  Executive  Office 
(Pfc'O)  ponfblio  response  to  this  guidance  is  the  ASC'E-V  T’M  jointly  maintained  ASC  Integrated 
Risk  Management  (IRM)  process. 

2.  To  ensure  Aircraft  Pfc'O  pngr-jns  meet  Air  Force  expectations  for  manage  incut  of  risk, 
program  managers  shall: 

a.  Have  a  documented  Risk  Management  Plan  ( RMP >  developed  by  a  cross-functional  team 
at  the  start  ol  their  program  and  updated  annually. 

b.  Complete  an  annual  Integrated  Risk  Assessment  (IRA),  ideally  m  conjunction  with  he 
annual  program  life  cycle  cost  esiimute,  lu  ensure  lisksAisk  hardling  are  quantified  ar.d 
appropriate  I  >  budgeted. 

c.  Develop  risk  lundlin.ymitigation  plans  that  track  the  highest  rises  identified  during  the 
TRA  process:  incorporate  those  risk  hi  noling  ,irategies  into  program  plans:  und  monitor  risk 
abatement  on  a  conlinux  s  basis. 

3.  Detailed  information  on  the  ASC  IRM  process  can  he  found  ul  hitps:.(Avww.ea.  wpafb.at.mil/ 

and  hi  l r  s :  •  ■' fi nance .  wpafb.af.mil.-'asi  tin  TM t  MC'.HTM.  ASC'EN  and  ASC/1 MC  web  ;sages. 

Additionally,  training  m  IKM  and  IRA  can  be  provided  upor.  rcqtest- 

4.  I  his  policy  memorancun  superseces  all  previous  ASG'CC  policy  on  IRM.  (Juestior.s  car.  be 
directed  lr  Mr.  Charles  1  ib.an,  ASC/ENSL  (937)  255-4002,  or  Mr.  Earl  Kcsstngcr.  ASC/FMCR, 
.937)  656-5473. 


-  Incorporate  mitigation  into  program 
plans 


JOHN  L.  HUDSON 
Lieutenant  General,  USAT 
Commander 


“IRM  ...  integrates  the  technical,  schedule,  and  cost  impacts  of  risk  areas  into  a 
complete  picture ....” 


Integrated  Risk  Assessment 
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Evaluation 


Analyses 


#123ABOEngine  Thrust  Requirements 


Owner 

jLtCol  Hodgkiss/Harrison 


Probability  of  Risk  Occurrence 
No  Risk 
r  Level  1:0-  20% 

Ip3! Level  2:  21-40% 

Level  3:  41  -  60% 

W  Level  4:  61  -  80% 

Level  5:  81  -  100% 

Consequence  of  Risk  Occurrence 


Risk  Description 


If  engine  performance  is  less  than  required,  then 
engine  redesign  will  have  to  occur,  causing  a  most 
|likely  2-mo  schedule  slip  and  $1.5M  cost  overrun. 
Annotations 


(Critical) 


t~  Negligible 
f~  Minor 
W  Moderate 
I-  Serious 
V  Critical 


Schedule  Link  | 


If  the  risk  occurs,  the  following  numbers  will  apply: 
COST: 

Best  case:  $0.9M 
Most  likely:  J1.5M 

Worst  case:  $2.3M  (assume  standing  army) 

SCHEDULE: 

Best  case:  0.5  mo 
Most  likely:  2.0  mo 

Worst  case:  4.0  mo  (assume  standing  army) 

Tied  to  line  item  #456  in  the  200-line  rollup 
Integrated  Master  Schedule  dated  9  Sep  06. 


\ 


Output: 

Risk  Matrix 
Schedule 
Cost  Estimate 
List  of  Risks 


Updated  compilation  of  (C/S/P)  risks 


P/CS 


Risks: 

•Performance 

•Schedule 

•Cost 

•Others 


Impacts 
•Performance 
•Schedule 
•Cost 
[•  Min 
*  •  ML 


Date:  8/31/2006  1 :36:52  PM 
Samples:  5000 
Unique  ID:  1 
Name:  ASIP  Program 


Completion  Std  Deviation:  27.55  days 
95%  Confidence  Interval:  0.76  days 
Each  bar  represents  10  days 


Risk+@TS^Crystal  Ball® 


'radeoffs  — 


Integrated  Risk  Assessment 


Presentations 
Other  Info 
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Risk  Handling 
Risk  Monitoring 


(IRA) 


Mj.ummif.nj.ui 

Edit  Preferences  View  Run  Help 
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10,000  Trials 
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Completion  Date 


1.0 
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Completion  Probability  Table 

0.8  ^ 

Prob 

Date 
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0.7  _q 

0.05 

9/9/08 
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11/17/08 
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9/23/08 
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0.15 

10/2/08 
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11/26/08 

04  | 

0.20 

10/9/08 

0.70 

12/3/08 
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10/16/08 

0.75 

12/9/08 

0.2  | 

0.30 

10/23/08 
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12/16/08 
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0.35 

10/28/08 
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12/23/08 
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0.40 

11/3/08 

0.90 

1/1/09 

0.45 

11/6/08 

0.95 

1/14/09 

0.50 

11/11/08 

1.00 

3/30/09 

.000 

$4,568,674 

►  [-Infinity 


$5,500,427  $6,432,179  $7,363,932 

Certainty  |90.06  %  4  BE] 


$8,295,685 


probability  distribution  -  total  program  cost 


probability  distribution  -  program  completion  date 


PROBABILITY 


Risk  Summary  (IRA) 
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Near 

Certain  ® 


Highly 
Likely  ^ 


Likely  3 


Unlikely  2 


Remote  -\ 
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Negligible  Minor  Moderate  Serious  Critical 


3.  Sensor  Qualification  Testing 

4.  U-2  Accreditation 
6.  Site  2  Accreditation 

15.  U-2  Timing  Source  Stability 

18.  Edwards  Connectivity  to  Site  2 

30.  GH  SAR  Antenna  Positions 

33.  ASIP  EMSEC  Test  on  Flight  Test/Production  A/C 

43.  Certified  DL  Encryptor  not  Available  for  Fielding 

45.  FARSITE  Support 

47.  Sensor/DCGS  Interface  Testing 

48.  GH  Ku/Band  5  Corrective  Action  Effectiveness 

50.  GH  Payload  Weight 

51.  Flight  Test  Sortie  Rate 


IMPACT 


2  risks  have  greatest  potential  to  impact  schedule: 
#30  GH  SAR  Antenna  Positions 
#51  Flight  Test  Sortie  Rate 
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ASIP  SoS  Risk  Management 
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Jeopardize  SoS  level 
performance  or  event 


Require  Government  or 
associate  contractor  effort 


Raytheon 


communications 


NORTHROP  GRUMMAN 
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ASIP  Risk  Management  Process  Model 
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Best  Practices 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Conduct  monthly  risk  review  boards 

-  PMs  &  CEs  actively  involved 

•  Include  mitigation  activities  in  IMS 

•  Brief  status  regularly  to  senior  leaders 

•  Carefully  describe  risks 

-  If  (root  cause),  then  (bad  outcome) 

•  Address  contractor  &  government  risks 

•  Include  step-downs  in  mitigation  plans 


Outstanding 


UCI 


PROBABILITY 


ASIP  Results 
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Take  Aways 

Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


•  Risks  are  present  in  every  program 

-  Can’t  afford  no/low  risk  programs 

-  May  be  acceptable  if  provides  opportunity 


Risk  management  is  proactive 

-  Determines  where/when  to  use  resources 

-  Necessary  to  make  acquisition  programs 
executable 


“If  you  don’t  actively  attack  the  risks,  they  will 
actively  attack  you.” 

~  Barry  Boehm  in  Software  Risk  Management 


/  cannot  imagine  any  conditions  which 
would  cause  a  ship  to  founder. 


DAU  -  Risk  Management  in  Systems  Engineering 


Captain  E.J.  Smith,  1906 
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Questions/Comments 


Dominant  Air  Power:  Design  For  Tomorrow... Deliver  Today 


Headquarters  U.S.  Air  Force 
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System,  and  System-of-Systems  Definition 
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Historical  Perspective 
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Pre-Acquisition  SE 
(“Pre-A  Systems  Thinking”) 

Overview 


■  Where  It’s  Required 

■  What  It  Is  (and  Is  Not) 

■  Key  Attributes 

■  Universal 

■  Collaborative 

■  Not  for  the  neophyte 

■  Responsive  but  realistic 

■  Smart  choices 

■  Why  It’s  Important 

■  The  Road  Ahead  ... 


Integrity  -  Service  -  Excellence 
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Concept  Refinement  Phase  -^-Technology  development  F  hase 


System  Development  &  Demonstration  Phase 
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Pre-Acquisition  “Systems  Thinking 

Where  It’s  Required 
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SE  needed  in  two  places 

During  development  of  all 
concepts  that  feed  an  AoA 
A  more-or-less  standard  set  of  process 
steps,  to  bound  the  problem 

On  selected  concept  from  the  AoA 
(can  be  spirals/increments  to' 
existing  programs) 
Leads  to  the  TDS  and  initial  SEP 
for  the  selected  concept 


Pre-Acquisition  “ Systems  Thinking” 

Informing  the  Decision-Making  Process 


What  it  is: 

■  Linkage  between  JCIDS  and  the  AoA 

■  A  disciplined  process  to: 

■  Scope  capability  needs 

■  Develop  concepts 

■  Do  necessary  groundwork  for  a  successful  AoA 

■  Essentially  a  method  to  develop  AoA  entry  criteria 

■  A  means  to  identify  candidate  solutions  and  assess 
their  TRLs 

■  Basis  for  Technology  Development  Strategy  (TDS) 

■  TDS  should  make  up  ~75%  of  content  of  SEP  submitted  at 
Milestone  /  Key  Decision  Point  A  for  selected  concept 
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Pre-Acquisition  “ Systems  Thinking” 

Informing  the  Decision-Making  Process 


Alternate  view: 

■  “Analysis  of  Problem”  as  precursor  to 
formal  AoA 

■  Methodology  that  uses  SE  processes  to 
translate  capability  statements  into  families  of 
concept  designs/approaches 

>  Trade  study  process 

>  Key  ground  rules  /  constraints 

>  Decision  criteria 

>  Methodology  for  populating  knowledge  base 

■  Describes  how  operational  context 
(architectures,  military  utility,  etc.)  drives  these 
translations 
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Pre-Acquisition  “ Systems  Thinking” 

Informing  the  Decision-Making  Process 


What  it  is  not: 

■  An  actual  requirement  development  effort 
under  JCIDS 

■  An  actual  AoA 

■  "Gaming  the  system"  in  favor  of  a 
particular  or  pre-determ ined  solution 


Integrity  -  Service  -  Excellence 
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Attributes 


■  Universal 

■  Collaborative 

■  Not  for  the  neophyte 

■  Responsive  but  realistic 

■  Smart  choices 
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Universality 


■  Applies  to  all  domains,  industries, 
product  areas,  research  areas  ... 


■  One  size  (policy,  process,  procedure, 
prior  idea  ...)  seldom  fits  all 
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Expanding  the  “V 


Figure  adapted  from  NDIA  Modeling  &  Simulation 
Committee  Final  Report  to  OUSD  (AT&L),  Mar  2004 


System , 
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Collaboration 


■  Understand  the  realities  of  -  and 
constraints  imposed  by  --  external 
factors  and  influences  across 
government,  industry,  academia 


■  The  human  is  an  external  factor,  and 
always  introduces  uncertainties 


Integrity  -  Service  -  Excellence 
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SE  for  SoS 
Challenges 


■  Unique  management  and  governance  issues 

■  Assets  acquired  /  operated  under  disparate  systems  and  policies 

■  Allocation  of  requirements  to  constituent  systems 

■  Integration  /  Verification 

■  Defining  architectures  to  link  systems  and  platforms 

■  Resource  constraints  on  physical  testing  drive  extensive  M&S 

■  Experimentation  as  a  development  tool 

■  Relatively  ad  hoc  configurations  in  operational  environment 

■  Legacy  system  modifications  /  updates 

■  Proprietary  issues 

■  Less-than-open  subsystem  and  component  designs 

■  Measurement 

■  Difficult  to  quantify  non-functional  requirements 

■  Mission-related  quality  attributes  (interoperability,  security,  etc.) 
largely  depend  on  architecture 
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Thinking 


■  Know  what  you  want, 
and  measure  smartly ... 

Accuracy  ^  Precision 


■  Beware  of  becoming  “DRIP” 
Data-Rich,  Information-Poor 
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Table  entries  (values  are  notional): 

0  -  not  applicable 

1  -  low 

2  -  nominal 

3  -  high 


Leading  Indicators 
Value  by  Life  Cycle  Phase 
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Candidate  Metrics  for  the 
Concept  Development  Process 


■  Distribution  of  concepts  in  the  development 
process  pipeline 

■  Number  of  items  in  each  of  the  various  stages  of  a 
concept’s  lifespan 

■  Concept  relevance 

■  How  well  a  set  of  concepts  addresses  the  cost  / 
performance  /  schedule  trade  space  for  a  specific 
shortfall 

■  Baseline  concept  schedule 

■  Progress  of  efforts  to  develop  relevant  and  mature 
concepts  to  meet  a  shortfall 
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Candidate  Metrics  for 
Development  of  a  Concept 


■  Supporting  analyses 

■  Cost 

■  Risk 

■  Military  Utility 

■  Other 

■  Technology  suitability 

■  Producibility 


■  Technical  progress 

■  Node  analysis 

■  System-  and  subsystem-level  trades 

■  Key  reviews 

■  Acquisition  strategy 


■  Transition  opportunities 
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Rational 


■  Customers/users  often  press 
for  immediate  solutions  over 
rigorous  process 


■  “Then  a  miracle  occurs” 
cannot  be  an  acquisition  or 
transition  strategy 
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SE  for  a  Product  or  System 

Transforming  Requirements  to  Design 
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‘Systems  Thinking ”  for  a  Capability 

Transforming  Needs  to  Requirements 
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Decisive 


■  Decomposition  and  allocation  must 
focus  on  HW,  SW,  or  human  first;  this 
decision  is  a  huge  driver  in  defining 
the  rest  of  the  solution  trade  space 


■  Do  it  right,  do  it  early;  do  it  early, 
do  it  right:  Systems  Engineering 
follows  -  but  must  NOT  replace  -- 
Systems  Thinking 
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How  We  Try  to  Fit  10  Lb.  of 
W  PROGRAM  Into  a  5  Lb.  BASELINE 
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Percent  of  Baseline  LCC  Incurred 
Percent  of  Baseline  LCC  Committed 
—  —  Cost  to  Identify  &  Resolve  a  Defect,  and  Incorporate  Change 


888S 


Why  It’s  Important 

Early  Decisions  Are  Key  Cost  Drivers 


Cumulative  LCC 
Cost  to  Fix 
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I  CBM  Life  Cycle  Cost,  1973 
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Top  10  Considerations  for  Applying 
ystems  Thinking  Early  in  the  Life  Cycle 


■  Applies  to  all  domains,  industries,  product  areas,  research  areas  ... 

■  One  size  (policy,  process,  procedure,  prior  idea  ...)  seldom  fits  all 

■  Understand  the  realities  of  -  and  constraints  imposed  by  -  external 
factors  and  influences  across  government,  industry,  academia 

■  The  human  is  an  external  factor,  and  always  introduces  uncertainties 

■  Know  what  you  want  and  measure  smartly  ...  Accuracy  /  Precision 

■  Beware  of  becoming  “DRIP”  -  Data-Rich,  Information-Poor 

■  Customers  often  press  for  immediate  solutions  over  rigorous  process 

■  “Then  a  miracle  occurs”  cannot  be  an  acquisition  or  transition  strategy 

■  Decomposition  and  allocation  can  focus  on  either  hw  or  sw  first;  this 
decision  is  a  huge  driver  in  defining  the  rest  of  the  solution  trade  space 

■  Do  it  right,  do  it  early;  do  it  early,  do  it  right:  Systems  Engineering 
must  follow  -  but  must  NOT  replace  -  Systems  Thinking 

UL  TIMA  TE  RESUL  TS 

>  Better  technical  planning,  better  integrated 

>  More  confidence  in  programs  entering  acquisition 
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How  NOT  to  do  Concept  Development 
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BACKUPS 
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Status  of  Current  AF  Efforts 


■  SMC  pilot  ongoing 

■  Three  drafts  of  process  guide  completed 

■  Tailored  Space  Situational  Awareness  capability  need  statement; 
conducted  exploratory  trades  and  initial  architecting 

■  Currently  in  design  phase  for  three  concepts  (one  ground-based, 
two  space-based);  cost  &  Military  Utility  analyses  ECD  30  Oct 

■  Initial  “Concept  Engineering  Plan”  (ConEP)  completed  for  each 

■  Proposing  policy  language  to  insert  AF  Chief  Engineer 
review  of  concept  pedigrees  as  AoA  “entry  criteria” 

■  NOT  an  in-depth  technical  review 

■  Provides  avenue  to  weed  out  “back-of-the-napkin”  concepts  early 

■  ASC  process  guide  in  work;  AAC  &  ESC  pilots  start  CY08 

FUTURE  STATE 

>  Rigorous  yet  adaptable  concept  development  processes  across  AF 

>  More  robust  concepts  going  into  AoAs 
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Pre-Acquisition  “Systems  Thinking” 

Boundary  Conditions 

Pre-Acquisition  SE  efforts,  like  those  throughout  the  rest 
of  the  life  cycle,  are  essentially  an  “integrating  function” 

■  Pre-A  SE  mainly  occurs  in  two  domains,  each  with  set 
boundaries 

>  The  first  domain  spans  the  period 
from  JCIDS  initiation  of  a  need  to 
AoA  entrance: 

>  The  second  domain  continues  the 
SE  functions  after  the  AoA  until 
formal  program  handoff: 

■  The  SE  functions  in  both  domains  are  fundamentally 
similar,  but  there  are  attributes  unique  to  each 


/AoA  Entrance 

F^SEjdSE 

DS 

Program  Initiation 

F2(SE)dSE 

AoA  Exit 
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Pre-Acquisition  “Systems  Thinking” 

Example 


Capability  need:  “Get  people  and  equipment 

across  a  body  of  water” 

■  First  pass  asks  key  questions: 

■  What  does  “water”  mean?  (Solution  sets  will  be  very  different 
for  Piscataway  Creek,  the  Potomac  River,  and  the  Pacific  Ocean.) 

■  Are  there  any  obvious  constraints?  (Sensitivity  to  water 
exposure?  Time-in-transit  limitations?) 

■  Initial  analysis  should  yield  various  methods,  and  a  cost  / 
risk  summary  for  each 

■  Airlift  ■  Drive  around  (depends  on 

■  Bridge  total  distance,  thus  time) 

■  Catapult  (unsuitable  for  people)  ■  Ferry 

■  Drive  across  (depends  on  ■  Helicopter 

depth,  current,  etc.)  ■  Tunnel 

■  Analysts  should  also  be  able  to  quickly  rule  out  candidates 
that  don’t  meet  constraints 
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Pre-Acquisition  “Systems  Thinking” 

Example 

■  Parametric  trades  within  a  method  (bridge,  tunnel,  etc.) 
consider  how  relevant  factors  (depth,  width,  current,  etc.) 
affect  a  baseline  candidate  solution 

■  “A  mile  upstream  the  channel  is  narrower.  The 
shorter  span  means  -30%  less  material  cost,  but 
road  access  and  construction  staging  are  difficult.” 

■  “A  mile  downstream  the  current  is  slower.  The 
longer  span  means  -20%  more  material  cost,  but 
you  can  complete  construction  earlier.” 

■  Once  the  AoA  looks  at  families  of  candidates  and  concludes 
that  a  bridge  is  the  best  solution,  a  similar  process  is 
employed  to  determine  the  optimum  type  (cantilever, 
suspension,  pontoon,  single-  or  two-span  draw,  etc.) 

■  Pre-AoA  measures  are  high-level  programmatic  /  operational 
parameters  (cost,  schedule,  vehicle  capacity,  etc.) 

■  Post-AoA  measures  have  a  more  traditional  design  and 
execution  focus  (EVM,  weight,  material  durability,  etc.) 


Reference 

location 
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Focus  Areas  for  SE  Planning 

Based  on  OSD  SEP  Preparation  Guide 


■  Program  Requirements 

■  Capabilities,  CONOPS,  KPPs 

■  Statutory/regulatory 

■  Specified/derived  performance 

■  Certifications 

■  Design  considerations 

■  Technical  Staffing/Organization 

■  Technical  authority 

■  Chief/Lead  Systems  Engineer 

■  IPT  coordination 

■  IPT  organization 

■  Organizational  depth 

■  Systems  Engineering  Process 

■  Technical  processes 

■  Technical  management  processes 

■  Process  improvements 

■  Key  tools  and  resources 

■  Trade  studies 

■  Linkage  to  contractor  SE  effort 


■  Technical  Baseline  Management 

■  Responsibilities 

■  Definition  of  baselines 

■  Requirements  traceability 

■  Specification  tree  and  WBS  link 

■  Technology  maturity  and  risk 

■  Technical  Review  Planning 

■  Event-driven  reviews 

■  Management  of  reviews 

■  Technical  authority  chair 

■  Key  stakeholder  participation 

■  Peer  participation 

■  Integration  with  Overall  Management 
of  the  Program 

■  Linkage  with  other  program  plans 

■  Program  manager’s  role  in  tech,  reviews 

■  Risk  management  integration 

■  Test  and  logistics  integration 

■  Contracting  considerations 


Highlight  -  greatest  applicability  to  Pre-A  efforts 
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Top  Considerations  for 
Applying  Early  SE  to  SoS 

■  An  end  product  that  is  usable  as  an  individual  entity  (e.g.,  by  s/n) 
is  generally  at  the  top  level  of  the  system  architecture.  An  end 
product  or  capability  that  incorporates  or  requires  multiple 
entities,  many  or  all  of  which  have  human  interfaces,  is  more  of 
an  SoS. 

■  The  whole  is  not  necessarily  equal  to  the  sum  of  the  parts.  What 
distinguishes  a  system  of  systems  from  a  discrete  system  is  that 
the  behavior  of  the  whole  cannot  be  predicted  from  the  aggregate 
of  the  constituent  elements  or  subsystems.  The  existence  of 
multiple  human  interactions  /  interfaces  is  a  huge  part  of  this. 

■  Integration  and  verification  plans  and  resources  must  be  in  place 
early.  This  includes  models  and  simulations,  experimentation 
venues,  and  integration  labs,  as  well  as  the  physical  assets  to  be 
tested.  However,  when  analyzing  test  data,  it  is  essential  to 
remember  that  if  enough  is  good,  more  is  not  necessarily  better. 
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ETER  VALUE 


Focus  Areas  for  Technical  Execution 


■  Representative  parameters  related  to 
Technical  Performance  Measures  (TPM) 

■  Hardware  -  weight,  speed,  power, 
cooling,  cross-section,  bandwidth 

■  Software  -  throughput,  lines  of  code 

■  Verification  -  test  asset  deliveries,  test 
points  completed  with  valid  data 

■  Logistics  -  reliability,  maintainability 

■  Integration  -  physical  and  information 
interface  definitions;  verification  plans 


Earned  Value  Management 
System  (EVMS)  data 

■  Cost  variances 

■  Schedule  variances 

Program  execution 

■  Staffing 

■  Subcontracting 

■  Specification  approvals 

■  Closure  of  review  actions 


Plan  is  probablyachievable_ Overly  optimistic  "get-  well"  plan 


Integrity  -  Service  -  Excellence 
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Emerging  Focus  Areas 


■  Technical 

■  SE  for  SoS  /  Architecting 

■  Manufacturing  Readiness 

■  Human  Systems  Integration 

■  Specifications  and  Standards 

■  Governance  &  Oversight 

■  MDA  Certification 

■  System  &  Software  Assurance  (Security  &  Program  Protection) 

■  Multi-Faceted 

■  Enterprise-level  SE 

■  Industrial  Base 


Integrity  -  Service  -  Excellence 
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SE  Perspectives 

Acquisition ,  Operations ,  Integration ,  Architecture 


OUSD(A  T&L)  /  JCS 
COCOMs 


SAF/AQ,  SAF/US, 
PEOs 

MAJCOMs 


Weapon  System  CEs 
&  Tech  Staff 


Operators  & 
Mai  ntai  tiers 


Project  Engineers 
(Program  &  Contractor) 

Logistics  Centers 


Supplier/  OEM 

Supply  Chain  Mgmt 


Capability 

Planning 


Technology  A  System  Development  A  Production  &  Deployment 
Development  &  Demonstration  Operations  &  Support 

Disposal 


Views  of  the  “universe 

Acquisition  Operational 


Test  &  integration  focus  (notional) 

DT&E  M&S  /  Experimentation 


OT&E 


Architecture  views 

(spans  are  not  authoritative) 


Integrity  -  Service  -  Excellence 
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Joint  Mission  Environment 
Test  Capability 
(JMETC) 


Briefing  for  the  Tenth  Annual 
Systems  Engineering  Conference 

Mr.  Richard  Lockhart 
Deputy  Director, 

Test  Resource  Management  Center 

October  24,  2007 
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TRMC  Functions 


Biennial  10-Year 
Strategic  Plannii 


Administer 
T&E  Investment 
Programs 

(CTEIP,  T&E/S&T, 
JMETC) 


DoD  Field  Activity 
Direct  Report  to  USD(AT&L) 
SES  Director 


Annual  T&E  Budget 
Certification 
(Military  Departments 
and  DoD  Agencies) 


Oversee  T&E  Budgets 
and  Infrastructure 
(MRTFB  and 
other  test  facilities' 


Sec.  231,  FY  2003  National  Defense  Authorization  Act 
DoD  Directive  5105.71,  March  8,  2004 


UNCLASSIFIED 
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Outline 


•  Background 


•  Program  Overview 


•  FY07  Accomplishments 


•  FY08  Plan 
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BACKGROUND 
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Testing  in  a  Joint  Environment 

Background 

•  March  2004  -  SPG:  “Joint  Testing  in  Force  Transformation” 

-  Policy  -  Developing  and  fielding  joint  force  capabilities  requires  adequate , 
realistic  test  and  evaluation  in  a  joint  operational  context 

-  Direction  -  DoD  will  provide  new  testing  capabilities  and  institutionalize  the 
evaluation  of  joint  system  effectiveness 

-  Action  -  DOT&E  lead  development  of  a  Roadmap  to  define  changes  to 
ensure  that  T&E  is  conducted  in  a  joint  environment  and  facilitates  the 
fielding  of  joint  capabilities 

•  November  2004  -  DEPSECDEF  approved  Roadmap,  validated  SPG 

-  Roadmap  identified  actions  to  implement  Testing  in  a  Joint  Environment, 
including 

•  Strengthen  and  enforce  DoD  policy  (DoDD  5000,  CJCSI  3170,  JCIDS)  to  elevate 
Joint  testing  requirements  in  DoD  acquisition 

•  Develop  Joint  testing  processes  and  methodology 

•  Establish  a  corporate  approach  to  Joint  distributed  testing  capabilities 

“...a  persistent,  mbustmodem  networking  infmstivcture  forsystems  engineeririg,  DT&E,  and \ 

OT&E...must  be  developed  that  connects  distributed  live,  virtual,  constructive  (LVC)  resources”  I 


Testing  in  a  Joir#  Environment  Roadmap,  dated  12  Nov  2004  UNCLASSIFIED 


Testing  in  a  Joint  Environment 
Background  -  continued 


•  December  2005  PB  07  PPM 

-  Approved  Joint  Mission  Environment  Test  Capability  (JMETC) 
program  to  provide: 

•  A  corporate  approach  to  joint  distributed  testing  capabilities 

-  Established  PE  under  AT&L  /  TRMC  for  execution 

•  October  2006 

-  JMETC  Program  Management  Office  established  in  Crystal  City, 
VA 


JMETC  IS  ONE  YEAR  OLD 
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Interoperability  /  Net-Ready  KPP 
Testing  Requirement 


“It  is  expected  any  resultant  materiel  solution  will  be  verified  through  testing  conducted  in  the  expected 

joint  operational  environment  to  demonstrate  joint  interoperability  and,  when  appropriate,  net-readiness” 

CJCSI  31 70.01  F,  dated  1  May  2007 

•  DoD  Policy  requires  Joint  interoperability  and  net- 
readiness  testing  during  acquisition 

•  Interoperability  and  Net-Ready  KPP  testing  requires 
testing  interactions  of  multiple  systems  at  the  same  time 

-  Systems  or  their  representations  are  not  all  co-located 

-  Need  to  test  early  and  throughout  system  development  process 

•  Transition  to  the  GIG  to  realize  Net-Centric  Warfare  will 
increase  the  requirement  for  interoperability  and,  thus, 
increase  the  need  for  distributed  testing 
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Interoperability  /  Net-Ready  KPP 

Testing  Problem 

•  Cost  prohibitive,  and  sometimes  impossible,  to  locate  all  required 
systems  in  one  place 

-  Laboratory  and  simulated  representations  may  be  the  only  assets  available 

-  Systems  and  their  representations  are  distributed  throughout  the  U.S. 
(industry,  test  ranges,  government  laboratories) 

•  Difficult,  time-consuming,  and  expensive  to  plan  and  execute 
distributed  test  events 

-  Networks  require  time-consuming  security  agreements  to  be  coordinated 

-  Instrumentation  data  definitions  differ  from  laboratory  to  laboratory 

-  Lack  of  universal  tools  complicates  test  integration 

-  Distributed  test  events  require  engineering  each  and  every  time 


Interoperability  and  Net-Ready  KPP  difficult  to  test 
extensively  or  early  in  acquisition 
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JMETC  PROGRAM 
OVERVIEW 
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What  is  JMETC? 


•  A  corporate  approach  for  linking  distributed  facilities 

-  Enables  customers  to  efficiently  evaluate  their  warfighting 
capabilities  in  a  joint  context 

-  Provides  compatibility  between  test  and  training 


•  A  core,  reusable,  and  easily  reconfigurable  infrastructure 

-  Consists  of  the  following  products: 

•  Persistent  connectivity 

•  Middleware 

•  Standard  interface  definitions  and  software  algorithms 

•  Distributed  test  support  tools 

•  Data  management  solution 

•  Reuse  repository 


•  Provides  customer  support  team  for  JMETC  products  and 
distributed  testing 
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JMETC  Supports: 


•  Testing  across  full  spectrum  of  acquisition  process 

-  Developmental  Test,  Operational  Test 

-  Interoperability  Certification 

-  Net-Ready  KPP  compliance 

•  Joint  mission  portfolio  testing 

•  Evaluation  of  weapons  systems  in  joint  mission  environment 

•  Conduct  of  live,  virtual  or  constructive  testing 

•  Conduct  of  joint  testing  and  training 

Used  whenever  you  need  to  link  resources  together 
to  conduct  a  distributed  test  event 
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JMETC  Infrastructure  (1  of  2) 


•  Persistent  connectivity 

-  Readily  available  network  configured  for  exchanging  test  data  over 
existing  DoD  data  transport  capabilities 

-  Solution:  Initial  VPN  has  been  established  and  is  operational  on  the 
Secure  Defense  Research  and  Engineering  Network 

•  Standards  for  Components  /  Interfaces 

-  A  collection  of  interface  definitions  and  software  algorithms  (e.g.,  Radar, 
TSPI,  coordinate  conversions,  unit  conversions,  etc.)  that  provide  a 
common  language  used  in  data  exchanges  between  systems 

-  Solution:  Use  TENA  and  upgrade  per  requirements  from  Users  Group 

•  Middleware 

-  Universal  data  distribution  software  used  by  every  node  to  send  and 
receive  data 

-  Solution:  Use  TENA  and  provide  gateways  to  connect  to  other  data 
protocols 
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JMETC  Infrastructure  (2  of  2) 


•  Distributed  Test  Support  Tools 

-  A  collection  of  common  software  applications  that  assist  test  engineers 
to  plan,  prepare,  set-up,  check-out,  monitor,  and  analyze  the  distributed 
test  event 

-  Solution:  JMETC  will  do  a  best  of  breed  search  for  test  support  tools 
with  final  recommendations  made  by  the  JMETC  Users  Group 


•  Data  Management  Solutions 

-  A  suite  of  data  archiving  solutions  to  store  test  data  collected  at  multiple 
locations  enabling  efficient  data  collection  and  analysis  for  events 

-  Solution:  CTEIP  study  to  develop  roadmap  in  FY  08  with  follow-on 
CTEIP  project  to  develop  solutions 


*  Reuse  Repository 

-  An  on-line  web  portal  with  relevant  distributed  event  information  (latest 
middleware,  software  components,  documentation,  lessons  learned, 
meta-data)  and  web-enabled  collaboration  services 

-  Solution:  Establishment  of  www.imetc.org  for  re-use  repository  in  FY08 

15  UNCLASSIFIED 
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Range 
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Definitions 
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Virtual 

Prototype 


TENA 

Standard 
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Definitions 


TENA 

Common 

Middleware 


JMETC 


Repository 


Infrastructure 


Distributed  Test  Data  Management 

Support  Tools  Solutions 


JMETC  Customer 


•  Program  Manager  (PM) 

-  Examples  include:  Acquisition  Program  Managers,  Portfolio  Managers, 
Advanced  Concept  Technology  Demonstration  (ACTD)  Managers,  etc. 

•Test  Agent 

-  Organizations  designated  by  PMs  to  lead  their  event  test  planning  and 
execution  (e.g.  White  Sands  Missile  Range  (WSMR)  and  Edwards  AFB) 

•  Resource  Owners  (Owners  of  Test  Resources) 

-  Capabilities  owned  across  the  Department  and  in  industry  that  test 
warfighting  capabilities  (e.g.,  Air  Combat  Environment  Test  &  Evaluation 
Facility  (ACETEF)) 

-  Test  Resources  include:  simulations,  measurement  facilities,  System 
Integration  Labs,  Hardware-in-the-Loop  Labs,  installed  systems  test 
facilities,  open  air  ranges 
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JMETC  Customer  Support 


|  JMETC  supports  customer  needs  at  customer  request 

Technical  Expertise: 

•  Assists  in  understanding  how  to  use  JMETC  products 

•  Assists  in  developing  T&E  strategy  and  requirements 

•  Supports  event  planning,  preparation,  and  execution 


Product  Support: 

•  Reviews  and  certifies  JMETC  products  for  corporate  use 

•  Integrates  new  nodes  onto  JMETC  VPN  with  security  agreements 

•  Augments  DREN  with  sites  critical  for  joint  testing  (maximizing  reuse) 

•  Measures  JMETC  infrastructure  performance 

•  Provides  Help  Desk  to  assist  JMETC  product  users 

•  Provides  semi-annual  TENA  training  classes 


Prioritization  of  effort  is  based  on  funding  available 
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Customer  Responsibility 


•  The  customer  is  responsible  for: 

-  Defining  requirements 

-  Providing  test  facilities  and  resources 

-  Installing  TENA  in  their  test  facilities  and  resources 

-  Requesting  and  funding  field  assistance: 

o  Technical  integration  support,  including  site  verification 
o  JMETC  product  training 

o  Detailed  event  planning,  preparation,  and  execution 

-  VPN  usage  fees  (charges  coordinated  with  JMETC  Program) 

-  Unique  middleware,  object  model,  and  software  tool 
development  and  upgrades 

*  Sites  not  on  JMETC  VPN  build  plan  may  fund  their  own 
addition  to  JMETC  infrastructure 
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Industry  Involvement 


•  Two  ways  to  participate  in  the  JMETC  infrastructure: 

-  Being  on  government  contract  to  support  a  program  or  test  event  using 
JMETC 

o  Contractor-funded  sites  possible  depending  on  priorities  and  resources 

-  Participate  in  the  JMETC  Users  Group 

o  Next  meeting  tentative  for  January  2008,  location  TBD 

•  TENA  Architecture  Management  Team  (AMT) 

-  Technical  forum  providing  open  dialogue  between  users  and  TENA 
developers 

o  Next  meeting  tentative  for  January  2008,  location  TBD 

-  Used  to  identify  issues,  vet  concerns,  debate  solutions,  and  agree  on 
way  forward 

-  Over  27  companies  currently  are  members  of  TENA  AMT 

-  TENA  middleware/object  models  freely  available  (www.tena-sda.org) 

Industry  is  a  key  component  in  a  successful  DoD  “corporate 
approach ”  to  linking  distributed  facilities 
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JMETC  Leadership  &  Governance 


JMETC 

Chain  of  Command 
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JMETC  Benefits 

•  Provides  Department-wide  capability  for: 

-  Evaluation  of  a  weapon  system  in  a  joint  context 

-  DT,  OT,  Interoperability  Certification,  Net-Ready  KPP  compliance 
testing,  Joint  Mission  Capability  Portfolio  testing,  etc. 

•  Provides  test  capability  aligned  with  JNTC 

-  Both  use  TENA  architecture 

-  Enables  joint  test  and  training 

•  Reduces  time  and  cost  by  providing 

-  Readily  available,  persistent  connectivity  with 
standing  network  security  agreements 

-  Common  integration  software  for  linking  sites 

-  Distributed  test  planning  support  tools 

•  Provides  distributed  test  expertise 
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FY  07  JMETC  ACCOMPLISHMENTS 


23 


UNCLASSIFIED 


JMETC  Accomplishments  -  FY07 

Summary  ( 1  of  2} 


•  Established  JMETC  Program  Office  October  2006 

•  Completed  the  Program  Execution  Guide  briefing  to  assist 
customers 

•  Conducted  a  DoD  Distributed  Test  Infrastructure 
Assessment  (requested  by  Joint  Staff  J8) 

•  Initiated  development  of  the  JMETC  Reuse  Repository 

•  Established  JMETC  Advisory  Group 

•  Held  regular  meetings  to  discuss  activities 

•  Established  JMETC  Users  Group 

•  First  Meeting,  Jun  19-20  with  140  attendees 

•  SIAP,  JSF,  and  FCS  briefed  their  plans 

•  Second  Meeting,  14-15  Aug  with  150  attendees 

•  Navy  DEP  briefed  their  plans 

•  Focus  groups  established  for: 

•  User  Requirements,  Tools,  InterTEC  Spiral  2,  Networks,  and  Data  Standards 
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JMETC  Accomplishments  -  FY07 

Summary  (2  of  2) 


•  Initiated  collaboration  with  the  Training  community 

•  Used  the  JNTC-sponsored  network  aggregator  in  first  test  event 
supported  by  JMETC 

•  Initiated  effort  to  peer  JNTC  JTEN  with  JMETC  VPN 

•  Established  the  JNTC  JATTL  as  a  beta-test  site  for  next  version  of 
TENA  (TENA  6.0  will  be  release  in  FY08) 

•  Supported  the  JFCOM  LVC  Architecture  Roadmap  Study 

•  Stood  up  the  JMETC  VPN  on  the  SDREN 

•  Established  8  locations  on  the  JMETC  VPN  available  for  future  use 

•  Pax  River,  Eglin,  White  Sands,  Redstone,  China  Lake,  Pt.  Mugu,  Pt. 
Loma,  and  JITC 

•  Supported  two  distributed  test  events 

•  Integral  Fire  07 

•  InterTEC  Spiral  2  Build  1 
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JMETC  FY  07  Accomplishments 

Integral  Fire  07  Test  Event 

Integral  Fire  07  Description: 

•  A  combined,  distributed  test  event  conducted  in  August  07  supporting 
the  following  three  customers: 

-  JFCOM  JSIC  JCAS  Assessment 

-  JTEM  Methodology  Assessment 

-  USAF  Warplan-Warfighter  Forwarder  (WWF) 

JMETC  Responsibilities: 

•  Overall  lead  for  creating  the  distributed  test  Infrastructure  including 
JMETC  VPN  (5  locations) 

•  Connect  three  enclaves  (total  of  15  locations)  using  the  JFCOM 
aggregator  router 

•  Conduct  systems  integration,  site  surveys,  and  dry  runs 

•  Oversee  operation  of  the  network  and  data  flow  among  all  sites  during 
the  event 

JMETC  Significant  Accomplishments: 

•  Stood  up  and  successfully  demonstrated  the  JMETC  VPN  within  90  days 

•  Successfully  used  the  Aggregation  Router  to  link  three  enclaves 

•  Supported  three  customers  conducting  tests  using  the  same  network  in 

the  same  time  frame  26  unclassified 


Integral  Fire  Infrastructure 


Existing  Connections 
used  by  JNTC  (not  to 
be  used  in  07-03) 


AIC  Pax  River 
Network  Monitoring 
of  JCAS  Enclave 


Pax  River 


E1  =  TacLane  with  JCAS  Key 
E2  =  TacLane  with  AF-ICE  Key 
E3  =  TacLane  with  SDREN  Key 


Aggregation  Router 


AIC  Pax  River 
Network  Monitoring 
of  JMETC  Enclave 


SDREN 
Key 


WARCAP, 

E2 

Pentagon 

x  e9 

— ISIMAF,  WP  AFB| 

z 

"lAOC,  Langley  AFB 


AV-8B  Lab 
China  Lake 


GWEF 
EglinAFB 


F/A-18  Lab 

IRCC 

China  Lake 

WSMR 

46TW 
Eglin  AFB 
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JMETC  FY  07  Accomplishments 
InterTEC  Spiral  2,  Build  1  Test  Event 

InterTEC  Description: 

•  Interoperability  T&E  Capability  (InterTEC)  is  an  OSD-sponsored,  Navy- 
led  project  under  the  Central  T&E  Investment  Program  (CTEIP) 

•  Purpose  is  to  develop  an  accredited  test  capability  to  conduct  joint 
interoperability  certification  and  joint  mission  thread  testing 


-  Spiral  2,  Build  1  Objectives: 

•  Developing  and  assessing  tools  to  test  joint  threads 

•  Assessing  the  C2  messages  sent  from  sensors  to  shooters  through 
command  and  control  systems  (GCCS-J,  GCCS-M,  GCCS-A,  ana 
TBMCS) 

-  JMETC  Responsibilities: 

•  Overall  lead  for  creating  the  Infrastructure  integrating  6  locations 

•  Conduct  systems  integration,  site  surveys,  and  dry  runs  in  preparation 
for  the  event 

•  Oversee  operation  of  the  network  and  data  flow  among  all  sites  during 
the  event 


-  JMETC  Significant  accomplishments 

•  Established  the  new  locations  on  the  JMETC  VPN  within  90  days 

•  Demonstrated  re-use  (three  locations  from  Integral  Fire  07  test) 

•  Successfully  used  the  Aggregation  Router 
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JMETC  Support  for 
InterTEC  Spiral  2  Build  1 
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JMETC  FY  08  PLAN 
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FY  08  Plan 


•  Complete  hiring  for  government  positions 

•  Publish  Program  Execution  Guide  (handbook) 

•  Expand  JMETC  Infrastructure 

-  Expand  the  JMETC  VPN  per  customer  requirements  and  potential  for  reuse 

-  Add  18  locations  for  a  total  of  26  available  for  reuse  by  the  end  of  FY  08 

•  Initiate  JMETC  Reuse  Repository  at  www.imetc.org 

•  Hold  quarterly  JMETC  Users  Group  and  JMETC  Advisory  Group  meetings 

•  Publish  Newsletter 

•  Collaboration  with  Training  Community 

-  Continue  to  collaborate  on  common  distributed  test  and  training  infrastructure 
requirements 

-  Continue  to  support  the  JFCOM  led  LVC  Architecture  Roadmap  Study 

-  Demonstrate  JTEN  and  JMETC  VPN  peering  capabilities 
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FY  08  Plan 

(continued) 


Support  Other  JM ETC-related  Activities 

-  JTEM  JT&E 

-  Support  3  studies  resulting  from  the  Distributed  Test  Infrastructure  Assessment 

•  Transitioning  Test  Capabilities  to  Internet  Protocol  version  6  (IPv6) 

•  Determining  the  Applicability  of  a  SOA  to  Support  Distributed  Testing 

•  Determining  Test  Infrastructure  Needed  to  Test  Warfighting  Capabilities  Using  the  GIG 


FY  08  Event  Support 

-  InterTEC  Spiral  2,  Build  2  and  System  Acceptance  Test  (SAT) 

•  Spiral  2,  Build  2  scheduled  in  April/May  08  followed  by  the  SAT  in  June  08 

•  Test  OTH-G  messages  using  a  Joint  Fires  Scenario 

•  Integrating  12  locations 

•  May  include  CVN-21  participation 

-  FCS  Combined  Test  Organization 

•  Experiment  and  test  of  the  infrastructure  needed  to  evaluate  joint  functionality  of  FCS 

•  Jun-Aug  08  (tentative) 

•  Planning  to  adhere  to  JTEM  Methods  and  Processes 

-  SIAP 

•  Risk  reduction  test  for  a  planned  Oct  08  event 
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JMETC  VPN  Locations  (FY07-08) 


Pt.  Mugu  (2) 


Camp 
Pendleton 


Pt.  Loma  (2) 


T 


AWACS,  Boeing  -  Seattle 


San  Diego 

FY07 

FY08 


Total  of  26  Sites 
Persistent  Capability 
Capable  of  REUSE  for  future  events 
VPN  transports  TENA,  tactical,  voice,  video 


V- 


— j 


China  Lake 


AWACS, 
Tinker  AFB 


'  ^Xscic, 

NGC  -  Newport  News 


,  Hanscom  AFB 


1  ,  Pax  River 

Dahlgren 
JSIC,  JFCOM 
Dam  Neck 


WSMR, 

JITC, 

Fort 

Huachuca 


Rivet  Joint, 

Raytheon 
Gre  nville 

JIL,  Lockheed,'^ 

Ft  Worth  v  Eg)jn  AFB 


Redstone  Arsenal, 

Huntsville 

SSC,  Charleston 


X) 


JSTARS, 

Melbourne 


CTSF,  Fort  Hood 


x 


33 


UNCLASSIFIED 


Summary 


•  JMETC  Program  Office  stood  up 

•  JMETC  VPN  established  -  26  locations  available  for  reuse 
by  the  end  of  FY  08  based  on  customer  requirements 

•  Successfully  supported  two  test  events  in  the  first  year 

•  Coordinating  with  JFCOM  to  bridge  test  and  training 
capabilities 

•  Collaborative  effort  with  the  Services  and  Industry 

•  Multiple  programs  requesting  support 

-  SIAP,  FCS,  CVN-21,  JSF,  MMA 

JMETC  IS  THE  CORPORATE  SOLUTION  FOR  JOINT 
DISTRIBUTED  TESTING  AND  IS  AVAILABLE  NOW 


34 


UNCLASSIFIED 


Points  of  Contact 


Program  Manager: 


Systems  Engineer: 


Chip  Ferguson 
703-604-0350  x138 
chip.ferguson@osd.mil 

George  Rumford 

703-601-5233 

george.rumford@osd.mil 
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Maintaining  System  Viability  for  the  Long  Term 

Paladin/FAASV  Integrated  Management  (PIM) 


Engineering  Conference 


10th  Annual  NDIA  Systems 
San  Diego,  California 

Peter  D.  Henry 

BAE  Systems  Land  &  Armaments 
pete.henry@baesystems.com 


Daniel  P.  Malinowski 

PEO  Ground  Combat  Systems 

daniel.p.malinowski@us.army.mil 


Manohar  Maman 

PEO  Ground  Combat  Systems 

manu.maman@us.army.mil 


October  24,  2007 
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■  Ml  09  family  of  vehicles 

■  The  rise  of  sustainability/support  issues 

■  Synchronizing  goals 

■  Paladin/FAASV  Integrated  Management  (PIM) 

■  Project  organization 

■  Engineering  challenges 

■  Conclusion 
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BAE  SYSTEMS 


Ml 09  FOV  Evolution 


25  Caliber  “Short  Tube” 
Range  1 5/20  Km 


1950’s 
Technology  Still 
Resides  in  a  Large 


1978^ 

M109A2/A3 


1982 

M992 


1992 

M992A1 


RAM  &  Safety  Improvements 
A2-New  Build 
A3-Upgrade 

1992 

__  M109A5 

a,. 


LHR  Engine. 

XTG  41 1  -4  T ransmission 
Stacker  Removal 


•  GPS  Integration 

•  Improved  Engine  Fire  Extinguis 

•  Stowage  Improvements 

•  Up-Powered  APU 

October  24,  2007 


A4-NBC/RAM  Improvements 
A5-New  Cannon  (24/30  Km) 


Digital  Fire  Control  System 
Automated  Gun  Laying 
Onboard  ballistic 
computation 
Inertial/GPS  navigation 
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Changing  Environment 


■  Through  the  1990’s  the  expectation  was  that  Crusader  and  Re- 
Supply  vehicle  would  replace  the  Paladin/FAASV  by  2008 

■  Long-term  design  sustainment  of  the  Ml  09  FOV  was  not  required 

■  In  2002,  the  Future  Combat  Systems  Non-Line  of  Sight  Cannon 
(NLOS-C)  replaced  the  Crusader  in  Army  development  plans; 

Ml  09  family  was  still  expected  to  be  supplanted  by  NLOS-C 

■  Army  Decision  Point  41 .1  dictated  a  path  to  a  modular  force 
comprised  of  a  mix  of  current  force  and  future  force  components, 
with  platforms  viable  and  sustainable  through  2050 

■  Long-term  sustainment  of  Paladin  again  became  a  requirement 


October  24,  2007 
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BAE  SYSTEMS 


SPH  Distribution  Plan 


I 

2005 


M109A6  Paladin 
29  HBCT  Battalions 
10  Fires  Battalions 


FCS  BCT  Delivery 

„ _ 

r - 


1 

NLOS-C 

15  Battalions 

V7^ 

■  T' 

Paladin/FAASV 

14  HBCT  BN 

10  Fires  BN 

1  1 

2017  2020 


2031  2060 


•  Fully  Sustainable  Paladin/FAASV  Baseline  required  to  support  the  HBCT 


•  Must  be  Interoperable  With  Future  Force  -  Will  fight  together 


•  Must  keep  pace  with  Bradley  &  Abrams  -  maintain  operational  relevance 


Significant  challenges  with  obsolescence;  very  limited  growth  potential; 

On  the  verge  of  becoming  unsustainable 
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Trends  &  Drivers 

*  Downward  Readiness  Trend: 


Decreases  at  5  of  6  Location  For  Last  12  Months 


Total  Armv 

Averaae 

Europe 

mum 

HIU 

mSMiSZ 

i-mixnriri 

FY04-05 

93.1% 

92.7% 

95.2% 

93.4% 

92.6% 

89.0% 

92.3% 

Last  1 2  Mos 

90.7% 

94.1% 

92.3% 

91 .3% 

87.8% 

86.4% 

73.8% 

-  Data  Gathered  From  Logistics  Integrated 
Database  (AMSAA) 

Vehicle  Age  Versus  Maintenance  Costs 
and  Burden  (14  yrs  vs.  8  yrs) 


FI.  Slyvmrl  FI.  Hood  BTC 

94.7% 

91.1% 

90.4% 

93.7% 

90.5% 

88.6% 

-  73%  Increase  in  Maintenance  Costs 

-  142%  Increase  Maintenance  Burden 

-  Data  Gathered  From  SDC  at  Ft.  Stewart  & 
Ft.  Hood 


Lociiiiorj 

Ft.  Hood 

Ft.  Stewart  j 

14  yrs 

8  yrs 

I*d  U  tu  i  L  iVc*  ilcic  l  Uii 

24 

14 

9.8 

7.2 

iXm  |'j  hjiju  UClcici 

$11,754 

$6,798 

235.2 

97.2 

FY01  FY02  FY03  FY04  FY05  FY06 
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BAE  SYSTEMS 


Sustainability: 

Paladin/FAASV  Component  Age 

Vehicle  Chassis  and  Major  Component  Designs  Over  45  Years  Old 
(TDP  developed  in  late  1950’s/early  1960’s) 

-  Vehicle  Design  Life  20  Years 

-  Ml 09  First  Fielded  in  1963 

•  All  M109A6  Paladins  Built  on  Refurbished 

Ml  09  Chassis 

"  Ml  09  Major  Component  Age 

Average  Age 

1.  Based  on  Paladin  Production  Data  at  York/LEAD 

2.  Based  on  Serial  Numbers  of  Chassis  Inducted  Into  LEAD  Production,  Analyzed 
Against  OEM  Production  Records  (A2)  &  Historical  Data  from  TACOM  (AO  &  , 


Calendar  Year 


> 

< 

<D 

D> 

CD 

(D 

> 

CO 

CD 


Cab  /  Paladin  Unique  Items  1 


1990’s  Design 

^(Post  Desert  Storm) 


Chassis  /  Re-Used  Parts 2  e.g. 

■  Chassis  Structure 

■  Transmission 

■  Road-Arms 

■  Final  Drives 

■  Rammer  /  Elevating  Cylinder 


1960’s  Design 

(Vietnam  Era) 


2006 

2025 

2050 

f^9 

28 

53 

<T  36 

55 

80 
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BAE  SYSTEMS 


Perspective 

■  Competing  priorities  have  limited  Army/OEM  investment  in  Paladin 

■  HBCT-centric  approach  brings  focus  &  visibility 

•  Three  legs  to  the  stool  -  Tanks,  Bradleys  &  Paladin 

•  Acknowledgement  that  like  Bradley  &  Abrams,  Paladin  will  be  in  the  fleet 
for  foreseeable  future 

■  Efforts  coming  together  -  positioning  program 

•  Dedicated  program  to  maintain  fleet  at  acceptable  average  age 

•  Formal  establishment  of  “Paladin  Integrated  Management”  (PIM)  line 

■  Sync  between  Combat  Developers,  Material  Developers  &  OEM 


October  24,  2007 
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BAE  SYSTEMS 


Prioritized  Goals 

■  PM  Priorities 

■  Support  the  fight 

•  Reset 

•  Excalibur 

■  Sustain  the  fleet 

•  PDFCS/APU/MACS  Retrofit 

•  RESET/RECAP 

•  Mitigate  Obsolescence 

■  Build  the  future 

•  Modularity  fieldings 

•  Develop  PIM  program 

•  Spin-out  /  tech  insertion 


TCM  Priorities 

■  Survivability 

■  Power  train 

■  Suspension 

■  Power  Management 

■  Digital  communications 
(cab  -  hull) 

■  Rammer  Improvements 
Vehicle  Health  Management 


Challenge:  convert  1-N  list  into  manageable  Army  program 


October  24,  2007 
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BAE  SYSTEMS 


Paladin  Integrated  Management  (PIM) 


Specific  program  &  plan 
to  address  long-term 
viability  of  Paladin 

Keyed  to  HBCT  (read 
Bradley)  commonality 

Leverages  FCS/NLOS 
technologies  as 
appropriate 


Paladin/FAASV  Integrated  Management 

(PIM) 


Process  That  Rebuilds  Platforms  to  Original  Factory  Standards,  Applies  Current  MWOs 
and  Delivers  “Like  New”  Platforms,  Which  Operate  with  Current  Technology 


Obtain  and  Maintain  a  Fleet  Age  of  10-12  Years 
Objectives 

-  Ensure  Supportability/Maintainability/Interoperability 

•  Leverage  Fleet  Commonality  for  Key  Components 

-  Engine/Transmission/Final  Drives/Suspension 

•  Replace  Obsolete  Components 

•  Reduce  Logistics  Footprint 

•  Reduce  Operations  &  Support  Costs 

•  Maintain  Performance 

•  Leverage  Abrams/Bradley  Improvements 

-  Improve  Crew  Survivability 

-  Technology  Insertion 

-  Managed  Through  a  Public  Private  Partnership  (P3) 


1  December  2006 


Process  That  Rebuilds  Platforms  to  Original  Factory  Standards,  Applies  Current  MWOs 
and  Delivers  “Like  New”  Platforms,  Which  Operate  with  Current  Technology 
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PIM  Strategy 

■  Many  Issues  are  Inter- Related;  Requires  Total  Weapon  System 
Approach  (vice  individual  efforts  to  solve  point  problems) 

■  PIM  Strategy  IAW  DP  41  (Viable  &  Sustainable  Platforms  beyond  2050) 

■  Provide  Viable  Life-Cycle  Solution  Beyond  2050 

■  Design,  Test,  and  Qualify  an  Affordable  Alternative  Structure  Around 
Selected  Components 

■  Current  Planning  Leverages  Commonality  With  HBCT  e.g. 

•  Bradley  Common  Track,  Engine,  Transmission,  etc 

•  Eliminate  Hydraulics  (Except  Recoil  System) 

•  Vehicle  Health  Management 

•  Reduces  Logistics  Footprint,  O&S  Costs  &  Development  Time/Cost 


Rebuilds  Platform,  Applies  Current  Modification  Work  Order’s  (MWO)  and  Delivers  a 

Ready,  Relevant  and  Sustainable  Platform 


October  24,  2007 
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PIM  Howitzer  Features 

Achieving  Sustainability  via  HBCT  Commonality 


BAE  SYSTEMS 


CREW  2 


Suspension  &Track 

-  6  Road  Arm  Stations  (BFV) 

-  Torsion  Bars  (BFV) 

-  4  Rotary  Dampers  (U2) 
-Track  19.1”  (BFV) 


Electronic  Systems 

- PDFCS 

-  DRU-H 

-  VHM 


Blue  Force  Tracking 

-  P3I  for  BFT 


Electrical  System 

-  600V,  70  kW  Integrated  Starter  /  Generator  (CMPS) 

-  600V  -  28V  Bi-Directional  conversion  (CMPS) 

-  Cable  Management  for  power  &  reliable  high  data 

transmission  capability  between  Cab  &  Chassis 


LEGEND 

Bradley  Common 

NLOS  common 
Common  Modular 
Power  System 
Paladin/FAASV 


ISCS  -  Individual/Spot  Coolin 
System  (improved  MCS) 


Gun  Drives 

-  Integrated  with  PDFCS 

-  600V  Electric  Elevation  drive  (NLOS) 

-  600V  Electric  Traverse  drive  (NLOS) 

-  Electric  Joysticks 

-  Manual  Gun  Drive  backups 


Chassis  (new  structure) 

-  Additional  ground  clearance 

-  Structure  integrity  (71500  lbs  GVW) 

-  Provisions  for  Mine  Blast  kit 
and  side  Armor 

Driver’s  Compartment 

-  Shift  Tower  (BFV) 

-  Brakes  (BFV) 

-  Steering  (BFV/Paladin) 

-  Seat  (BFV/Paladin) 

-  Hatch  -  larger  diameter  than  Paladin 

-  Composite  Armor 

-  Instrument  Panel  (BFV/M109  &  Digital  Display) 


Power  Train 

-  Engine  600  HP  (BFV) 

-  Transmission  HMPT  500-3ECB  (BFV) 

-  PTO  (upgraded  BFV-style) 

-  New  Cooling  system 

-  Engine  Compt  AFES  (FAASV) 

-  Final  drive  (BFV) 


COS  Cupola  TAGS 

Armament 

-  39  caliber/ 155  mm  (Paladin) 

-  Travel  Lock  (Paladin) 

-  600V  electric  rammer  (NLOS) 


October  24,  2007 
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PIM-FAASV  Features 

Maximal  commonality  with  PIM  Howitzer 


CREW  II 

ISCS  -  Individual/Spot  Cooling 
System  (improved  MCS) 


Electrical  System 

-  Common  Modular  Power  System  (CMPS)  incl  600V, 

70  kW  Integrated  Starter  /  Generator 

-  600V  -  28V  Bi-Directional  conversion 

Power  Train 

-  Engine  600  HP  (BFV) 

-  Transmission  HMPT  500-3ECB  (i 

-  PTO  (upgraded  BFV-style) 

-  New  Cooling  system 

-  Engine  Compt  AFES  (FAASV)  {r> 

-  Final  drive  (BFV) 

-  Easily  accessible  Air 

Cleaner  Filter 


Suspension  &Track 

-  6  Road  Arm  Stations  (BFV) 

-  Torsion  Bars  (BFV) 

-  4  Rotary  Dampers  (U2) 
-Track  19.1”  (BFV) 


Crew  Compartment 

-  Crew  seating  (FAASV) 

-  Rear  door  (FAASV) 

-  Crew  AFES  (FAASV) 

Blue  Force  Tracking 

-  P3I  for  BFT 

Chassis  (new  structure) 

Lower  Chassis  common  with  SPH 
Provisions  for  Mine  Blast  kit  &  Side  Armor 
Additional  ground  clearance 
Flat  Floor  in  rear 

Structure  integrity  (71500  lbs  GVW) 


Electronic  Systems 

-  Power  Management  (CMPS) 

-  VHM 


Driver  Compartment 

-  Shift  Tower  (BFV) 

-  Brakes  (BFV) 

-  Steering  (BFV/Paladin) 

-  Seat  (BFV/Paladin) 

-  Hatch  -  larger  diameter 

-  Composite  Armor  (Paladin) 

-  Instrument  Panel  (BFV/M109  &  Digital  Display) 


LEGEND 

Bradley  Common 

NLOS  common 
Common  Modular 
Power  System 
Paladin/FAASV 


Mission  Equipment 

-  Projectile  Racks  (FAASV) 

-  MACS  Stowage  (FAASV) 
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IR&D  Prototype  -  October  2007 


October  24,  2007 
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PIM  System  Development  Approach 


■  Total  system  approach  vs.  point  solutions  for  individual  problems 
(typical  STS  task  order-approach) 

■  Design  approach  is  that  of  a  Systems  Integration  problem  vs.  a 
development  problem  -  IPTs  to  use  HBCT-common  solution  where  one 
exists 

■  HBCT  commonality  of  subsystems  provides  lower  development  and 
acquisition  costs  than  a  new  unique  design 


Public-Private  Partnership:  Industry-Government  collaboration  with  common  goals  & 

objectives  sharing  successes  and  failures 


October  24,  2007 
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PIM  IPT  Hierarchy 


■  Each  IPT  is  jointly  chaired  by  Government  and  Industry  leads 

■  Core  and  ad  hoc  /  supporting  members  are  identified  in  IPT  charters 

■  IPT  Core  membership  includes  key  suppliers 


October  24,  2007 
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SE  Challenges  in  a  Sustainment  Project 


■  Baseline  Requirements  Set  may  be  Incomplete 

•  e.g.,  off-road  mobility  requirement  not  explicitly  defined 

■  User  can  Become  Accustomed  to  or  Reliant  on  Features 
that  are  not  Defined  in  the  Requirements  Baseline 

■  Design  Baseline  Documented  to  Old  Documentation 
Standards 

•  e.g.,  DOD-STD-1679  Software  Documentation 

•  e.g.,  Ada  Programming  Language 

■  Design  Baseline  Developed  and  Tested  using  Lower- 
Maturity  Processes  and  Standards 

■  Performance  baseline  developed  to  old  mission  profiles 

•  e.g.,  Fulda  Gap  vs.  SW  Asia 

•  May  Require  Updated  or  New  Mission  Profiles 


October  24,  2007 


©2007  BAE  Systems  Land  &  Armaments  L.P. 


17 


BAE  SYSTEMS 


Summary 


■  PIM  leverages  components,  systems  and  proven  technologies  available 
today  to  ensure  that  the  Paladin/FAASV  fleet  remains  ready,  relevant 
and  sustainable  beyond  2050 

■  HBCT  commonality  reduces  development,  acquisition  and  sustainment 
costs 

■  The  PIM  Public-Private  partnership  leverages  the  strengths  of  both 
public  and  private  sectors  in  an  open,  collaborative  process 


Partnering  for  the  Soldier 
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BAE  SYSTEMS 


Paladin  Enterprise  - 

Leveraging  Best  of  Public  &  Private  Sectors 


DEPOT 

WEAPONS  *  cmw VetKLES  ♦  AMMUNITION 


BAE  SYSTEMS 
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SRWAR 

▼ 

Systems  Center 
Charleston 


Executing  a  Successful  CMMI®  Maturity 
Level  3  SCAMPISM  for  SPAWAR  Systems 
Center  Charleston  (SSC-C) 

Michael  T.  Kutch,  Jr.  Mike  Knox 


SPAWAR  Systems  Center  Charleston  (SSC-C) 

Head,  Intelligence  &  Information  Warfare  Systems 
Engineering  Department 

National  Competency  Lead  for  I/A  5.8 

Deputy  National  Competency  Lead  for  ISR/IO  5.6 


Technical  Software  Services,  Inc. 
Director,  Implementation  and  Support 
SEI  Authorized  Instructor 


10th  Annual  NDIA  Systems  Engineering  Conference 

October  24,  2007 


Improving  operational  effectiveness  through  C4ISR  common  integrated  solutions 


mm' 


®  Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM  and  CMMI  are  registered  in  the  U.S.  Patent  &  Trademark  Office. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University 


N65236-ENGOPS-BRIEF-0046-1 .0 


Approved  for  public  release;  distribution  is  unlimited  (15  OCT  2007) 


Tei 


■  SEIftarmcr 


CHSOFT 


Technical  Software-  Services,  Inc. 


Presentation  Outline 


SRMfl 

¥ - 

Systems  Center 

Charleston 

> Background 
>Road  to  Maturity  Level  3 
> Appraisal  Planning/Execution 
>  Lessons  Learned 
> Beyond  Maturity  Level  3 


Te 


SEIParLoer 

CHSOFT 


Technical  SoftnArare  Services,  Inc. 


2 


Approved  for  public  release;  distribution  is  unlimited  (15  OCT  2007) 


SRWAR 

▼ 

Systems  Center 

Charleston 

Background 


Technical  Software-  Services,  Inc. 


N65236-ENGOPS-BRIEF-0046-1 .0 
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Where  We  Fit 


SP/MR 


Systems  Center 
Charleston 


SPAWAR 

Space  and  Naval  Warfare 
Systems  Command 

▼ 


NETWARCOM 

MARCOR 

-  ADDU  for  C4I  — 

NAVSEA 

1  NAVAIR 

V 


President 

non-DoD 

Secretary  of  Defense 


Secretary  of  the  Navy 


U---J 


Other  DoD 


1 - 

1 

_ 1 _ 

CNO 

ASN  (RDA) 

Fleet  Support 

Acquisition 

1 

— | - 

i 

1 

| 

1 

1 

SPAWAR 

NAVSEA 

NAVAIR 

NAVSUP 

NAVFAC 

San  Diego,  CA 

Washington,  DC  Patuxent  River,  MD  Washington,  DC 

Washington,  DC 

r  t 

SYSCEN  SYSCEN  SYSCEN  SFA 

San  Diego,  CA  New  Orleans,  LA  Norfolk,  VA  Chantilly,  VA 

v  v  y  v 


Tec 
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SE I  Partner 

iCHSOFT 

Technical  Software  Services,  Inc. 


What  We  Do 


Leveraging 

Technology 


if  i-tGHT  Speed 


smm 


Systems  Center 
Charleston 


Connecting  the  Warfighter 


Mission-  We  enable  knowledge  superiority 
to  Naval  and  Joint  Warfighters  through  the 
development,  acquisition,  and  life-cycle 
support  of  effective,  integrated  C4ISR 
Information 
Technology, 
and  Space 
capabilities. 

Vision- 
Fully  Netted 
in  Three 

We  are  the 
Principal  C4I 
Acquisition 
Engineering  & 

Integration 
Center  on  the 
East  Coast 
&  Principal 
C4ISR  ISEA  for 

the  Navy 


MWR-  MobileNet 


Body  Worn 
Variant 


NETCOP-Network  Common 
P  Operating  Picture 


Rapid 

Prototyping _ £ 


Speed  to 
Capability 


Te 


SEIRartner 

CHSOFT 


Connecting  the  Warfighter  to  the 
resources  needed  to  win  GWOT 


Technical  Software  Services,  Inc. 
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Who  We  Are 


SHMR 


Systems  Center 

Charleston 


A  Large  Systems  &  Software  Engineering  Organization 


Over  70%  of 
workforce  is  in  an 
engineering  or 
computer-related 
discipline 


■Contracts  &  Supply,  112 

■Finance  &  Budget,  77 

-General  Clerical,  51 
IT  Support,  76 
Other,  174 


Program  Mgmt,  106  /  \  Logistics,  79 


•The  solutions  to  the  global  war  on  terror  developed  by  SPAWAR 
result  from  good  systems  and  software  engineering 

•Systems  engineering  is  our  core  competency 
•Total  workforce  of  ~  2,300  employees 


Te 


SEIParLoer 

CHSOFT 


Technical  SoftiArare  Services,  Inc. 
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A  Vision  of  World  Class 


Systems  Center 
Charleston 


When  you  want  it  done  right, 
Who  do  you  want  working  on  it? 


Cutting  corners, 
undisciplined, 
untrained 


Rigorous  processes, 
Skilled  resources 


Permission  to  use  Redneck  Mechanic  photo  received  from  Dave  Lilligren,  3/9/2007 

Permission  to  use  NASCAR  Technical  Institute  photo  received  from  Popular  Mechanics,  3/16/2007 
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Process  Improvement  and 
Systems  Engineering  Strategy  -  2003 
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•  Vision 

-  Develop  and  maintain  a  World  Class  Systems  Engineering  Organization 

•  Approach 

-  Achieve  Command-wide  operational  consistency 

-  Based  on  ISO  15288  -  systems  engineering 

-  Based  on  ISO  12207  -  software  engineering 

-  Measure  using  best  practices  of  CMMI® 

•  Goals 

-  CMMI  Maturity  Level  2  by  April,  2005 

-  CMMI  Maturity  Level  3  by  April,  2007 


Both  Goals  attained  on  schedule 
1st  SPAWAR  Systems  Center  to  Achieve  ML2  and  ML3 
New  Goal:  Maturity  Level  4  by  2010 
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Critical  Success  Factors 
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CRITICAL  SUCCESS  FACTORS  FOR  SE  REVITALIZATION 

Command-wide  Policy 
(Create  vision  that  is  urgent) 

Assign  Responsibilities 
(Strong  Change  Agents  are  essential) 

Strategy  and  Plan  (Include 
knowledge  of  why  change  is 
necessary  and  benefits) 

Provide  Training 

Senior  Management  Support 

Build  a  Central  Repository 

Provide  Resources  and  Funding 
(New  Organizational  Structure 
Usually  Needed) 

Measure  and  Communicate  Progress 
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SSC-C  SE  Revitalization  Plan 
Aligned  with  DoD  SE  Revitalization 


i 

SSC-C  SE  Instruction 


SSC-C  SE 
Process  Manual 


SSC-C  SW-Dev 
Process  Manual 


SSC-C  SW-Maint 
Process  Manual 


EPO  Website 


ePIan  Builder 


Underway 

Completed/Ongoing 

_ ! _ 

Intro  to  PI  WBT 


SE  101  WBT 


SE  Fundamentals 


SE  for  Managers 

Project  &  Process 
Workshop _ 

Intro  to  Software  Engr. 


Architecture  Dev.  WBT 


Certification/Degrees 


_ i _ 

CMMI®  Level  2 


CMMI®  Level  3 


CMMI®  Level  4/5 


Project  Reviews 


Balanced  Scorecard 


Lean  Six  Sigma 


Integrated  Product 
Teams 


IT  Tools 
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Process  Improvement  Infrastructure: 

Organization 
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Timeline  2001-2002 
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•  Prior  to  2001 

-  Code  70  had  experience  with  SW-CMM® 

•2001 

-  SSC-C  Process  Improvement  (PI)  effort  began 

-  Code  70  developed  PI  Policy  for  SE,  SW,  and  Security 
Engineering  using  SEI  CMM®  and  CMMI® 

-  Code  70  Engineering  Process  Group  expanded  to  Command¬ 
wide 

-  Engineering  Process  Office  (EPO)  Website  started 

-  Pilot  Projects  selected  and  evaluated 

-  Some  templates  published 

•2002 

-  Began  developing  and  delivering  training 

-  Began  conducting  Class  “C”  assessments  as  progress  checks 
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Timeline  2003 
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Systems  Center 
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•2003 

-  Established  and  Funded  Dir.  of  Engineering  Operations  position 

•  Staffed  Engineering  Process  Office  (EPO) 

-  Developed  Organizational  Standard  Policies 

•  Policy  for  each  CMMI®  Level  2  and  3  Process  Area 

-  Developed  Organizational  Standard  Process  Manuals 

•  Top  Level 

•  Systems  Engineering 

•  Software  Development 

•  Software  Maintenance 

•  Supporting  Processes 

•  Process  Manual  for  each  CMMI®  Level  2  and  3  Process  Area 

-  Developed  plan  templates 

-  Coached  and  mentored  pilot  projects 

-  Built  tools 

-  Developed  and  delivered  training 

-  Performed  interim  assessments 
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Timeline  2004-2005 


spmuR 

Systems  Center 

Charleston 

•2004 

-  Conducted  project-level  Maturity  Level  (ML)  2  SCAMPISM  Class 
“A”  appraisals 

•  6  Projects  Appraised 

•  6  Achieved  ML2 


•  April  2005 

-  Conducted  Command-level  ML2  SCAMPISM  Class  “A”  appraisal  - 
First  SPAWAR  Systems  Center  to  achieve  Command-level  ML2 


World  Class 
Systems 
Engineering 

Capability 
Maturity 
Model 
Integration 
(CM  Ml®) 


Systems 

Center 

Charleston 


MATURITY  LEVEL 

N.  April  28,  2005 
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The  Second  Wave  -  ML2  to  ML31 
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•  Addressed  the  three  Organizational  Process  Areas 
early  to  provide  a  smoother  transition  to  ML3 

-  Organizational  Process  Focus  ( OPF }  -  Purpose:  Plan,  implement, 
and  deploy  organizational  process  improvements  based  on  an 
understanding  of  the  current  strengths  and  weaknesses. 

•  Determined  Process  Improvement  Opportunities 

-  Management  commitment  -  the  PI  strategy 

-  Benchmarked  current  state,  addressed  identified  needs/gaps 

•  Planned  and  Implemented  Process  Improvements 

-  Determined  Scope,  Model  (CMMI-SE/SW),  Approach  (Staged,  but 
appraise  using  Continuous) 

-  Created  appropriate  teams  to  champion  PI  efforts 

•  Deployed  Organizational  Process  Assets  and  Incorporated  Lessons 
Learned 

-  Shared  sample  project  plans,  improvements,  etc.,  across  the 
organization 
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The  Second  Wave  -  ML2  to  ML32 
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•  Addressed  the  three  Organizational  Process  Areas 
early  to  provide  a  smoother  transition  to  ML3  (con’t) 

-  Organizational  Process  Definition  ( OPD )  -  Purpose:  Establish 
and  maintain  a  usable  set  of  organizational  process  assets  and 
work  environment  standards. 

•  Developed  EPO  website,  which  is  a  repository  for  standard  process 
manuals,  SOPs,  checklists,  etc.  The  site  also  contains  Tailoring 
criteria  and  other  useful  resources  such  as  sample  plans,  etc., 
shared  with  the  SSC-C  organization  by  its  projects 

•  Built  SSC-C  Organizational  Measurement  Repository  (OMR)  for 
projects  to  use  Tor  managing  their  projects  and  capturing 
standardized  cost,  schedule,  and  process  performance" 
measurement  data 

-  Defined  Balanced  Scorecard  measures  directly  related  to  CMMI® 
and  Process  Improvement 
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The  Second  Wave  -  ML2  to  ML33 
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•  Addressed  the  three  Organizational  Process  Areas 
early  to  provide  a  smoother  transition  to  ML3  (con’t) 

-  Organizational  Training  ( OT )  -  Purpose:  Develop  the  skills  and 
knowledge  of  people  so  they  can  perform  their  roles  effectively 
and  efficiently. 

•  Identified  the  training  needed  by  the  organization 

•  Obtained  and  provided  training  to  address  those  needs 

•  Established  and  maintained  training  capability 

•  Established  and  maintained  training  records 

•  Assessed  training  effectiveness 

-  Objective  evaluation  of  OT  process  performed  by  the  Process  and 
Product  Quality  Integrated  Product  Team  (PPQA  IPT) 
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The  Second  Wave  -  ML2  to  ML34 
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•  SSC-C  organization  developed  basic  Tailoring  Guidelines 

•  SSC-C  Projects  developed  ML2-to-ML3  Action  Plans 

•  Developed  internal  “self-assessment”  process  for  measuring 
ongoing  implementation  of  ML2  processes 

•  Continued  enhancing  ePIan  Builder  tool  to  create  new  plans 
(e.g.,  SEP/SEMP)  that  are  ML3  compliant 

•  Updated/Improved  existing  plans 

•  Provided  additional  CMMI®  Training 

•  Added  Work  Breakdown  Structure  Tool  and  Architecture 
Development  Web-Based  Training  Course 

•  Continued  to  Measure  and  Communicate  Progress 

•  Maintained  Momentum  and  Commitment  to  Goals 
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Timeline  2005-2006 
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•  May  -  Dec  2005 

-  Updated  Organizational  processes  with  ML3  language 

-  Built  Organizational  Measurement  Repository  (OMR)  to  track 
cost,  schedule,  and  process  performance  measurement  data 

-  Developed  Sample  ML3  plans 

-  Projects:  Built  ML2  to  ML3  transition  plans 

•  Coaching  and  mentoring  continued 


•2006 

-  Conducted  project-level  Maturity  Level  3  SCAMPISM  Class  “A” 
appraisals 

•  6  Projects  Appraised  between  June  and  December 

•  5  Achieved  ML3 

-  Projects  worked  to  correct  consistent  weaknesses  in  Peer 
Reviews,  Decision  Analysis  and  Resolution  (DAR),  PPQA 
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Timeline  20071 
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•  January  2007 

-  1  additional  project  achieved  ML3 

-  Collected  data  from  30+  “non-focused”  projects 

•  Tailoring  Guidelines 

•  Project  Management  Plans 

•  SEMP/SDPs 

•  PPQA  Plans 

•  CM  Plans 

•  M&A  Plans 

•  February  2007 

-  Conducted  5-day  Readiness  Review 

-  Collected  additional  artifacts  needed 
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Timeline  20072 
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•  April  2007 

-  Conducted  Command-level  ML3  SCAMPISM  Class  “A”  appraisal  - 
First  SPAWAR  Systems  Center  to  achieve  Command-level  ML3 

-  9  Projects  in  appraisal  scope  -  7  Focused,  2  Non-Focused 

•  >8000  artifacts  submitted,  1 64  interviewees 

-  SEI  Senior  Member  was  Lead  Appraiser  (Team  Leader) 

-  2  other  SEI  Authorized  Leads  on  the  Team 

-  1  Government  person  from  NSA 

-  1  Government  person  from  SSC-C 

-  3  team  members  with  multi-appraisal 
experience 
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Success  Factors  of  Implementation1 
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•  Carefully  select  Initial  Projects 

-  Start  with  interested  projects 

•  High  Sponsor  interest 

•  Strong  need/desire  to  improve 


*  Set  Guidelines  (criteria)  that  yield  benefits,  for 
example,  SSC-C’s  CMMI®  Projects  meet  the  following: 


-  Systems  or  software  engineering  effort 

-  Funding  directly  with  SSC-C 

-  SSC-C  performs  the  Project  Management  function 

-  SSC-C  PM  is  directly  responsible  for  product  delivery 

-  Multi-year  effort 

-  Over  $2M  per  year 

-  Not  limited  to  level  of  effort  for  services 

-  Not  merely  a  pass-through  contract 
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Success  Factors  of  Implementation2 
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•  Assign  a  CMMI®  resource  to  each  project 

-  Strong  facilitator  with  strong  CMMI®  knowledge 

-  Conduct  regular  (at  least  monthly)  process-focused  meetings  to 
ensure  steady  progress 

•  Include  all  key  process  area  members  (including  contractors) 

-  Review  project’s  plans,  SOPs,  work  products 

-  Explain  process  area  practices  to  the  team’s  subject  matter 
experts 

•  Relates  model  to  project 

•  Helps  team  define  typical  work  products 

•  Helps  team  identify  and  collect  direct  and  indirect  evidence 

-  Conduct  mini  assessments  to  benchmark  progress 

-  Share/provide  organizational  tools,  templates 
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Success  Factors  of  Implementation3 
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•  Project  Team 

-  Project  Manager  -  involved  and  committed  to  success 

-  Document  Specialist/Technical  Writer  role  for  coordinating 
documentation,  revisions 

-  Active,  skilled  PPQA  manager  is  a  great  benefit 

•  Also  can  serve  as  the  Measurement  Analyst 

-  Useful  plans  are  built  by  the  key  players;  shelfware  is  built  by  the 
novice  or  new  contractor 

-  Don’t  let  one  person  wear  too  many  hats 

•  Resource  the  team  properly 

-  New  technology  and  complex  systems  are  NOT  necessary  for 
success 

•  A  Customer  that  supports  the  initiative  is  a  plus 
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•  Recognize  and  Publicize  Early  Successes 

-  ‘Project-level’  SCAMPIs  provided  early  successes  due  to 
conducting  the  appraisal  using  the  “continuous  representation”  of 
the  model 


•  Scope  of  appraisal  looked  at  all  7  ML2  PAs,  then  1 1  ML3  PAs 

•  If  all  the  PAs  were  satisfied,  then  the  project  achieved  ML2  and/or  ML3  through 
equivalent  staging 

•  Or,  Projects  received  Capability  Level  2/3  for  various  PAs  satisfied  (e.g.,  CM, 
SAM,  REQM,  PP,  PMC,  TS,  PI,  DAR) 

-  Led  to  BIG  success!  -  SSC-C  became  the  first  SPAWAR  Systems 
Center  to  achieve  CMMI®  Maturity  Level  2  (April  2005) 


-  Continued  similar  approach  to  Maturity  Level  3 

•  1st  Successful  ML3  Program  -  July  2006 

•  4  more  projects  achieved  ML3  in  late  2006 

-  Command  CMMI®  Maturity  Level  3  -  April,  2007 

•  1st  SPAWAR  Systems  Center  to  achieve  ML3 
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•  7  SEI  staff  members  were  involved  in  the  SSC-C 
Class  “A”  SCAMPIs 

•  Required  early  planning  to  get  each  SEI  staff 
member’s  commitment  to  appraisal  dates 

•  Built  detailed  schedule  for  ML2  and  ML3  project  and 
organizational-level  appraisals 

•  Obtained  commitment  from  project  team  members 
concerning  availability  on  appraisal  dates 

•  Reserved  conference  and  meeting  rooms  well  in 
advance 
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Appraisal  Execution1 
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•  Pre-Readiness  Reviews  (PRRs)  helped  to  ensure 
projects  were  ready  and  the  Formal  RR  would 
lead  to  90%-1 00%  coverage 

-  Used  Appraisal  tool  to  conduct  PRRs 

•  Provided  early  and  easy  access  to  the  direct  and  indirect 
evidence  for  each  process  area’s  specific  and  generic  practices 

•  Provided  means  for  communicating  appraisal  team  comments 
-  Used  convention  to  denote  status  of  each  practice 

(e.g.,  PRR-SG:  Direct  OE  satisfies  practice  OR 
PRR-SG:  Direct  and  indirect  OE  is  too  old) 

•  Provided  early  feedback  to  the  projects 

•  Provided  easy  upload  of  new  artifacts  supplied  by  projects 
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•  Formal  RRs  conducted  on-site  with  Appraisal 
Team  Members  (ATMs) 

-SEI  Lead  Appraiser  and  ATMs  worked  as  a  team 

-  Used  Appraisal  tool  to  conduct  RR 

•  Provided  easy  access  to  the  direct  and  indirect  evidence  for 
each  process  area’s  specific  and  generic  practices 

•  Provided  means  for  communicating  appraisal  team  comments 

-  Used  convention  to  denote  status  of  each  practice 

(e.g.  RR-CS:  Direct  OE  indicates  performance  of  practice 
OR  RR-CS:  Direct  and  indirect  OE  is  too  old) 

•  Provided  good  feedback  to  the  projects  on  items  still  missing 

•  Provided  easy  upload  of  new  artifacts  supplied  by  projects 
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•  SCAMPISM  Class  A  appraisals  conducted  on-site 

-Involved  mostly  the  “Interview”  process  since  RR 
ensured  direct  and  indirect  coverage  was  evident 

-  Used  Appraisal  tool  to  conduct  SCAMPISM 

•  Affirmation  section  of  tool  allowed  for  easy  update  following 
each  interview 


•  Tool  allowed  primary  team  member  to  select  practice 
compliance  and  secondary  member  to  concur  (or  not) 

•  Authorized  lead  appraiser  (team  lead)  then  verified  each 
practice  within  the  process  area 

•  Built-in  color  coding  provided  easy  visibility  to  “weaknesses” 

•  Facilitated  voting  process  at  Goal  level  and  Process  Area 

-Each  project-level  ML3  SCAMPISM  conducted  in  5 
days  and  Command-level  ML3  SCAMPISM  conducted 
in  1 0  days 
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Lessons  Learned  -  Implementation 
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•  Senior  Management  support  is  critical  to  success 

•  Training 

-  Everyone  needs  to  be  engaged  -  “train  the  masses” 

-  Specific  training  for  process  owners/subject  matter  experts 

•  Utilize  Teams  (IPTs)  as  champions  of  specific  processes 

-  Multi-department  representation 

-  Change  agent  mentality 

-  Process-focused  charters 

•  Resource  Properly 

-  Implement  with  projects  that  want  to  improve,  can  benefit  from  efforts, 
and  that  recognize  own  weaknesses 

-  EPO  staff  provided  skilled  coaching,  resources,  support,  and  tools 

-  Project  members  learned  by  doing  and  maintaining 


•  Goals  and  Publicity 

-  Keep  goals  to  sizable  bites  (projects) 

-  Publicize  successes;  Share  best  practices 
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•  Provide  CMMI®  mentoring  and  coaching  for  projects 
selected  for  an  appraisal 


•  Build  detailed  schedules  for  appraisals  early  in 
planning  phase  to  use  as  a  roadmap 


•  Plan  early  in  order  to  obtain  project  team  member  and 
appraisal  team  member  commitment  to  appraisal 
dates 
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Lessons  Learned  -  Appraisals2 
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•  Invest  in  an  Appraisal  Tool  to  facilitate  easy 
collection  and  evaluation  of  appraisal  data 


•  Perform  a  Pre-Readiness  Review  to  ensure  minimal 
coverage  gaps  are  identified  at  the  formal  Readiness 
Review 


*  Conduct  individual  project  appraisals  to  ensure 
successful  organizational  appraisals 


•  Document  Lessons  Learned  from  conducting 
appraisals  to  improve  the  appraisal  process 
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What  has  success  meant? 
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•  Business  Results 

-  SCN:  “They  see  us  as  a  model  and  want  to  increase  our  efforts.” 

-  Automation  Program:  “We  had  hundreds  of  sites  and  there  was  a 
need  for  a  structured  organization  to  put  a  ‘wrapper’  around  that 
and  control  it.  CMMI  became  the  wrapper.” 

-  CICS:  “CMMI  was  key  to  achieving  the  project  goal.” 

-  VIDS:  “The  VIDS  failure  (2000)  motivated  implementing  CMMI 
because  the  team  needed  to  change  course  or  the  customer 
would  have  no  confidence  in  system  development.  It  was  a 
tremendous  success...” 

•  Others  Asking  for  Help 

-  PMS  408  -  CREW  program 

-  SESG  /  NAVAIR  /  NAVSEA 

-  Marine  Corp  -  Quantico 

-  Air  Armament  Center,  Eglin  AFB 
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Beyond  Maturity  Level  3 


Plan  of  Action  for  ML4/5 
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Continue  Momentum 
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•  No  more  “Ratings  for  Life” 

-  Ratings  are  now  valid  for  only  3  years  (April  2007-  April 
2010) 

-  SSC-C  will  lose  its  CMMI®  ML3  rating  on  27  April  2010  if 
another  Command-level  SCAMPISM  Class  “A”  appraisal  is 
not  successfully  completed  before  then 

•  Sustain  the  current  Command-sponsored  projects 
(representative  sample) 

•  Seif-Assessments/Appraisals  -  mentoring  and  coaching  of  more 
projects 

•  Plan  for  and  Implement 

-  CMMI®  VI  .2  (CMMI®-DEV)  New  Model 

-  Maturity  Levels  4/5 
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Plan  of  Action  for  ML4/51 
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•  Take  a  fresh  look  at  the  entire  measurement  program  with 
an  eye  towards  managing  the  projects  using  quantitative 
data 


•  Collect  and  evaluate  project  historical  data  for  measuring 
cost,  schedule,  and  quality 

•  Establish  a  process  for  maintaining  the  appropriate  data 
to  begin  managing  quantitatively 

-  Select  at  least  one  “main  contributor”  sub  process  per  project 
lifecycle  phase,  at  least  one  project  management  sub  process 
and  at  least  one  support  sub  process 


*  Statistically  manage  the  data 


-  Using  statistical  methods  (e.g.,  Statistical  Process  Control  charts, 
histograms,  trend  charts,  etc.) 


Te 


SEIRarLoer 

CHSOFT 


Technical  SoftiArare  Services,  Inc. 


39 


Approved  for  public  release;  distribution  is  unlimited  (15  OCT  2007) 


Plan  of  Action  for  ML4/52 
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•  Demonstrate  stable  historical  data  for  measuring  cost, 
schedule,  and  quality 

-  Stable  data  will  help  you  answer  questions  like: 

•  Can  you  predict  where  your  next  data  point  will  fall? 

•  Do  you  know  what  your  baseline  is  for  cost/schedule 
performance? 

•  Is  your  product  quality  what  you  expect  it  to  be? 

•  Are  you  finding  “enough”  defects  before  the  customer  gets 
the  product? 

-  As  a  guideline,  strive  for  at  least  4  consecutive  data  points 
within  your  established  control  limits 
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Plan  of  Action  for  ML4/53 
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•  Formalize  performance  baselines  for  the  project  and 
provide  baseline  data  to  organization 


•  Re-establish  quantitative  objectives  (for  example): 

-  Reduce  cost  variance  to  +/-  5% 

-  Reduce  schedule  variance  to  +/- 10% 

-  Reduce  delivered  defects  by  +/- 1 0% 

-  Improve  major  saves  found  in  peer  reviews  by  20% 


•  Use  baselines  and  variance  to  predict  future 
performance 


*  Keep  up  the  ML2  and  ML3  process  performance! 
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•  May  -  Dec  2007 

-  Developed  Process  Improvement  Plan  for  ML4/5 

-  Developed  Detailed  Schedule  for  ML4/5 

-  Developed  QPM  Plan  Template 

-  Held  various  ML4  Meetings  with  projects 

-  Held  SCAMPISM  for  one  project  using  CMMI®  vl  .2 

•  September:  Project  achieved  ML3 

-  Increase  usage  of  tools  across  departments/projects 

-  Add  additional  plans  to  ePIan  Builder  as  needed 

-  Continue  internal  CMMI®  Level  3  mini  assessments 


Begin  Maturity  Level  4/5  implementation 
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Timeline  20072 
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•  May  -  Dec  2007  con’t 

-Enhance/Expand  OMR 

•  More  Quality  Data  from  Peer  Reviews,  Testing  Phase  and 
Defects  from  Production 

•  More  Statistical  Process  Control  (SPC)  Charts 

-Command  and  Department  Project  Reviews  process 

•  Look  at  quality  of  plans  and  implementation  of  best  practices 

•  Reviews  of  project  status  by  management  driven  by  project 
metrics 

•  More  Peer  Reviews  to  measure  “saves” 

-Better  tailoring  guidance  for  smaller  projects 


Begin  Maturity  Level  4/5  implementation 
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•2008 

-  Conduct  ML3  SCAMPISM  Class  “A”  appraisals  for  new  projects 

-  Conduct  ML4/5  SCAMPISM  Class  “A”  appraisal  for  one  program 

•2009 

-  Conduct  ML3  SCAMPISM  Class  “A”  appraisals  on  other  Command 
projects 

-  Conduct  ML4/5  SCAMPISM  Class  “A”  appraisals  on  other 
Command  projects 


•2010 


-  Conduct  SSC-C  Command-level  ML4  SCAMPISM  Class  “A” 
appraisal  in  April  201 0 
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•  Decided  on  Approach  -  Use  CM  Ml®  for  Process 
Improvement  and  Measuring  Progress 

•  Using  extensive  research,  determined  the  ‘Critical 
Success  Factors’  for  Implementing  CMMI® 

•  Built  Plan  of  Action/Detailed  Schedule  for  Appraisals 

•  Provided  Training  -  Systems  Engineering,  Processes,  & 
CMMI® 


•  Advertised  Early  Successes 

•  Implemented  Plan  Successfully  for  Phase  1  -  CMMI® 
Maturity  Level  2  and  Phase  2  -  CMMI®  Maturity  Level  3 

-  On  schedule,  on  budget 


•  Laying  groundwork  for  higher  maturity 
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Any  Questions? 


Contact  Information: 

Michael  T.  Kutch,  Jr. 

SPAWAR  Systems  Center  Charleston 
Email:  michael.kutch@navv.mil 
Phone:  843-218-5706 


Mike  Knox 
TECHSOFT,  Inc. 

Email:  miknox@techsoft.com 
Phone:  850-469-0086 
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What  is  Cyberspace? 


Cyberspace*  is  a  term  used  to  define  the  virtual  world,  built  entirely  of 
computers,  computer  networks,  and  associated  systems  around  the 
globe 


“Although  Cyberspace  would  not  exist  without  physics,  it  is  by  no 
means  bounded  to  the  pure  physical  reality  term.  ” 

Wertheim,  M.,  De  hemelpoort  van  cyberspace,  Anthos,  Amsterdam,  2000. 


*The  term  was  coined  by  William  Gibson  in  his  novel  Neuromancer 


-  Software  Engineering  Institute 


Achieving  Agility  in  Cyberspace 

Carnegie  Mellon  10/17/07 


2 


©  2007  Carnegie  Mellon  University 


Cyberspace  as  a  Theater  of  Engagement 


Loss  of  boundaries 

•  A  threat  can  arise  instantaneously  anywhere.  (SIPRNet  is  not  immune.) 

Fluidity  of  the  environment 

•  No  consistent  front  or  mode  of  attack 

No  global  visibility 

•  Large,  chaotic,  opaque  motives,  masking  identity  is  easy 

Uncertain  nature  of  time 

•  Not  necessarily  a  relation  between  the  time  an  attack  occurs  and  the 
time  it  was  launched 

Overlapping  and  shared  jurisdiction 

•  Involves  many  parties,  many  areas  have  no  clear  dominion,  spillover 
across  jurisdictions  is  the  norm 
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What  are  the  Military  Threats  in  Cyberspace?* 


Limited  cyber  war:  Information  infrastructure  is  the  means  and  target 
of  attack  (i.e.,  low-intensity  conflict) 

•  e.g.,  denial  of  service  attacks  using  botnets  against  Estonia  in  Spring, 
2007 

Unlimited  cyber  war:  Comprehensive  in  scope  and  target  coverage 
(i.e.,  high  intensity  conflict) 

•  no  distinctions  between  military  and  civilian  targets  or  between  the 
home  front  and  the  fighting  front. 

•  physical  consequences  and  casualties 

—  attacks  deliberately  intended  to  create  mayhem  and  destruction 

•  economic  and  social  impact — in  addition  to  the  loss  of  life — could  be 
profound 


NATO  Review,  Vol  49,  No  4,  Winter  2001 
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Framing  the  Cyberspace  Theater 


Predictability  of 
Threat  Situation 
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Low-Intensity,  High-Predictability  Threats 


Adversaries  threaten  (and  present  opportunities)  consistent  with  plans 

•  Goal  is  to  develop  tactics  that  counter  these  predictable  threats. 

•  For  the  most  part,  these  threats  can  be  addressed  by  good  hygiene, 
such  as 

—  installing  security  patches  and  procedures  in  a  timely  way 

—  verifying  compliance 

—  managing  passwords  and  other  data  securely 

—  monitoring  attempts  to  access  systems 

—  gathering  data  about  the  attackers  and  turning  attackers’  actions 


against  them 
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Low-Intensity,  Low-Predictability  Threats 


Adversaries  place  unanticipated  demands  on  the  organization: 

•  Malicious  agent  employs  a  novel  strategy,  exploits  a  new  flaw,  or 
targets  a  new  victim. 

•  Some  form  of  emergency  response  is  required. 

Activities  supporting  this  function  include: 

•  coordinating  the  response  to  counter  the  threat 

•  monitoring  the  frequency/type  of  events  managed  by  the  emergency 
response  capability 

•  identifying  the  chain  of  culpability,  where  possible 

•  analyzing  patterns  of  activity  in  order  to  understand  targets, 
motivations,  strategy,  and  tactics 


=^=  Software  Engineering  Institute 


Carnegie  Mellon 


Achieving  Agility  in  Cyberspace 
10/17/07 

©  2007  Carnegie  Mellon  University 


7 


High-Intensity,  High-Predictability  Threats 


Adversaries  use  high-intensity  but  predictable  attacks  to  achieve 
large-scale  geopolitical  or  economic  gain. 

Key  to  success  is  to  war-game — to  coordinate  relationships  with 
identified  partners  to  meet  anticipated  threats 

To  prepare  for  these  threats 

•  develop  scenarios  that  reflect  likely  forms  of  attack 

•  identify  external  partners  that  will  be  involved  and  establish  coordinated 
plans  for  responsibilities 

•  train  personnel  on  available  tools  and  technologies 

•  experiment  with  tools  and  tactics 

•  allow  sufficient  flexibility  to  allow  personnel  to  adapt  to  minor  variations 
of  known  situations 
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High-Intensity,  Low-Predictability  Threats 


High-intensity  and  low-predictability  conflict  implies 

•  The  good  hygiene  approach  (bottom  left  quadrant)  is  not 
sufficient  to  meet  the  demand  of  a  rapidly  changing  threat. 

•  Emergency  response  teams  (bottom  right  quadrant)  will  become 
overwhelmed  as  the  intensity  of  the  conflict  and  the  stakes 
involved  increase. 

•  War-gamed  responses  (top  left  quadrant)  are  unlikely  to  map 
beyond  the  opening  salvo  because  the  intelligent  adversary  will 
continually  adapt  to  the  response. 


No  matter  how  good  the  hygiene,  emergency  response,  and  war¬ 
gaming,  intelligent  adversaries  can  drive  the  situation  into  the  top 
right  quadrant  whenever  they  choose. 
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The  Cyberspace  Theater’s  Double  Challenge 


Intensity  of 
Threat 


Predictability  of 
Threat  Situation 


High 


High 

Intensity 


Low 

Intensity 


Low 


Mission-centric,  i 
war-game 

planning  / 

Threat-specific 

customization, 

orchestration, 

synchronization 

Basic  preventive 
care  with  good 
hygiene 

/  Pre-structured 
emergency 
/  response 

Dealing  with  unanticipated 
forms  of  threat 


Working 

across 

multiple 

enter¬ 

prises 
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Forms  of  Agility  Required 


Type  II  Agility 

Anticipate  the  demands  on  the 
mission  and  how  products  or 
services  will  be  used 

Multiple  organizations  brought 
under  a  unified  chain  of 


Predictability  of 
Threat  Situation 


Vcommand 


Type  I  Agility 

Anticipate  the  demands  on  the 
mission  of  defending  against 
intrusion 

Anticipate  how  products  or  services 
will  be  used 

Ensure  that  managerial  entities  apply 
\appropriate  commands 


Intensity  of 
Threat 


Type  III  Agility 

Can’t  anticipate  the 
demands  on  the  mission 

Can’t  anticipate  how 
products  or  services  will  be 
used 

Multiple  organizations  each 
with  its  own  chain  of 
command 


Type  I  Agility  + 
Contingency  Planning 


Source:  The  Three  Agilities, 

Philip  Boxer  &  Richard  Veryard, 
2006;  http://asymetricdesign.com/ 
archives/18 
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An  Unfortunate  Trend 
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How  Does  Agility  Relate  to  Command? 


Agility  Type 

Command  Governance 

Type  1 

■  within  the  enterprise 

■  to  predicted  threats 

Stretching  resources  across  the  organisation  to  optimally  meet 
demands  (i.e.,  cost  efficiency). 

Ensuring  that  rules  are  followed 

Type  II 

■  across  enterprises 

■  to  predicted  threats 

Leveraging  existing  infrastructure  and  capabilities  to  address  threats 

Acting  intelligently  by  capturing  and  driving  key  information  and 
knowledge  through  the  organization 

Co-ordinating  relationships  and  processes  between  multiple  players 
(i.e.,  flexibility). 

Type  III 

■  across  enterprises 

■  to  unpredictable 
threats 

Harmonizing  competing  priorities,  multiple  strategies,  and 
technologies  across  organizations 

Sensing  and  responding  across  organizationsjo  new  threats  and 
opportunities 

Shift  command  authority  to  the  edge 
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Distinguishing  Forms  of  Command 


The  nature  of  the  managerial  control  is* 

•  Directed 

—  Command  that  can  be  controlled  by  a  central  authority 

•  Directed  Collaboration 

—  Command  that  requires  collaboration  to  fulfill  an  agreed-upon  central 
purpose 

•  Distributed  Collaboration 

—  Command  where  there  is  no  centrally  agreed-upon  purpose 
(The  purpose  must  be  built  in  response  to  situations.) 


*  “Architecting  Principles  for  Systems  of  Systems,  ”  Mark  W.  Maier.  http://www.infoed.com/oDen/papers/svstems.htm 
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Mapping  Command  Types  to  Agility  Types 


Demands/ 

Purposes 


Anticipated  Unanticipated 


Multiple 

Directed 

Distributed 

Collaboration 

Collaboration 

Autonomous 

(Type  II  Agility) 

(Type  III  Agility) 

Command 

Entities 

Single 

Directed 

Directed 

Composition 

Composition 

(Type  1  Agility) 

(Type  1  Agility  + 
Contingency  Plan’g) 
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Distributed  Collaboration,  Type  III  Agility 
Requires  Edge-Synchronization 


This  means 

•  Missions  are  defined  at  the  edge  where  the  threat  is  encountered, 
rather  than  at  the  center. 

•  The  infrastructures  have  to  be  “loosely-coupled”  and  “under¬ 
constrained”  (i.e.,  able  to  be  orchestrated  and  composed  at  the  edge). 

This  in  turn  requires  us  to  develop 

•  command  structures  that  support  power-to-the-edge,  and 

•  agile  infrastructures — with  stratified  granularity — that  are  sufficiently 
expressive  to  enable  power-at-the-edge. 
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How  do  we  get  there? 


The  way  forward  - 1 

Individual  components  must  be  re-architected  (i)  to  remove 
semantic  coupling  that  constrains  the  way  components  can  be 
used,  (ii)  to  establish  requisite  granularity,  and  (iii)  to  support 
multiple  ways  in  which  they  can  be  fused  with  other  components 


The  way  forward  -  2 

Requisite  interoperability  must  be 
modeled  to  identify  risks 


Asynchronous 
tight  coupled 


Autonomous 

Command 

Entities 


Multiple 


Upgraded  tc 
provide  ability  to[ 
migrate  role 


Wide  range  of  role 
and  function  to 
support  cyber 
operations 


Upgrade  b; 
extending 
functionality 


Synchronous 
tight  coupled 


Anticipated  Unanticipated 

Demands/Purposes 


Extensible 
architecture  - 
asynchronous 
loose-coupled 

This  is  a  dead  end. 


It  is  not  possible  to  go 
directly  from  bottom-right  to 
top-right  because  the 
strongly  coupled  semantic 
relationships  and  component 
granularity  constrain  the 
degree  to  which  we  can  put 
pieces  together 
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Model  Interoperability  Through  the  Command 
Structures  and  infrastructures  in  Their  Contexts-of-Use 


Predictability  of 

Model  interoperability  with  5  layers  of  analysis: 

•  Structure/Function:  The  physical  structure  and 
functioning  of  resources  and  capabilities. 

•  Trace:  The  digital  processes  and  systems  that 
interact  with  the  physical  processes. 

•  Hierarchy:  The  formal  hierarchies  under  which 
the  uses  made  of  both  the  physical  and  the  digital 
are  held  accountable. 

•  Synchronization:  The  lateral  relations  of  synchronization 
and  orchestration  within  and  between  the  organizations 
providing  services  “on  the  ground” 

•  Demand:  The  nature  of  the  contexts-of-use  giving  rise  to 
demands  on  the  way  the  operations  are  organized  to 
deliver  services  effectively  and  timely. 

These  5  layers  combine  to  form  a  model  of  the  operational  space  as  a  whole,  enabling 
Cyber  Command  to  analyse  the  threats  associated  with  orchestrating  and  synchronizing 
systems  of  systems  in  relation  to  particular  forms  of  demand. 
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Visual  PAN — Rapid,  Well  Structured, 
Spaghetti 


The  PAN  symbols  and  their  relationship  rules  generate  five  interlocking 
layers  in  the  visual  model. 


hd- 

■S-  -atf®'  (™ 

I _ _ _ r 


Structure/Function 


A 


Demand 


NATO  UNCLASSIFIED 


Source:  An  Examination  of  a  Structural 
Modeling  Risk  Probe  Technique,  Anderson, 
Boxer  &  Brownsword  (2006), _ 


Software  Engineering  Institute  CarnegieMellon 


„  http://www.  sei.cmu.edu/publications/docum 
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Stratification  Brings  Structure  to  the  Spaghetti 


A  six-layer  stratification  forms  a  framework  against  which  the  people, 
processes,  and  technical  structures  are  analyzed  in  relation  to  the  demands 
being  placed  upon  them. 


mission  problem 


Source:  An  Examination  of  a  Structural 
Modeling  Risk  Probe  Technique, 
Anderson,  Boxer  &  Brownsword  (2006), 
http://www.sei.cmu.edu/publications/doc 
uments/06.reports/06sr01 7.html 


orchestrations  of  constituent 
capabilities  e.g.  of  datalink,  esm 


outcomes  e.g.  certified 
mods,  on  station 

know-how  e.g.  programmers, 
test  design 

services  e.g.  display  consoles, 
mission  planning 

processes  e.g.  change 
notifications,  iff 


Underlying 

infrastructures 


super-  dirLv^__ 
structure  organisation 
e.g.  it  wing,  e.g.  ops 
iamco,  wing,  data 
shape  management 


£nts 
e.g.  nav 
output, 
identity 
tracks 


constituent 
capabilities 
e.g.  comms 
interoperability 


composition  of 
orchestrated 
constituent 
capabilities 


,  sources  of 


Stratification  Layers 

□  6  -  Mission  environments 

□  5  -  Operational  performance  of  the  capability 

□  4  -  Orchestration  of  capabilities  by  crew  and  operators 

□  3  -  Activities  supporting  the  operational  capability 

□  2  -  Activity  chains  involved  in  integrating  components 

□  1  -  Services,  systems,  and  know-how 


_ repair 
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Type  0  -  Constructive  Risk  Landscape 


11 

10 

9 


operator 


The  Constructive  Risk  Landscape  reveals  the 
degree  of  isolation  between  the  many  structural 
entities  in  this  system  of  systems. 


NATO  UNCLASSIFIED 


Low  q  values  indicate  isolation 


data 
management 


q  =  number  of  events  related  to  service 

k  =  number  of  other  services  with  common  events  at  this  level  of  q 
^SourcerAn^ExBmination  of  a  Structurat  Modeling  Risk  Probe  Technique,  Anderson,  Boxer  &  Brownsword 

(2^^^http://www. sei.cmu.edu/publications/documents/06.reports/06sr01 7.html  Achieving  Agility  in  Cyberspace 
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©  2007  Carnegie  Mellon  University 


23 


Type  I  -  Customization  Risk  Landscape 


The  Customization  Landscape  reveals  islands  of  high  connectivity 
with  broad  regions  of  separation. 


yQotffnp'  An  Examination  of  a  Structural  Modeling  Risk  Probe  Technique,  Anderson,  Boxer  &  Brownsword  Achieving  Agility  in  Cyberspace 

A  lei  loll  10/17/07 
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Type  II  -  Orchestration  Risk  Landscape 


The  Orchestration  Landscape  reveals  areas  of  isolation,  islands 
of  high  connectivity,  and  broad  regions  of  separation. 


Source:  AnExamination  of  a  Structural  Modeling  Risk  Probe  Technique,  Anderson,  Boxer  &  Brownsword 

http://www.  sei.  cmu.edu/publications/documents/06.  reports/06sr0 1 7. html 
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Type  III  -  Synchronization  Risk  Landscape 


The  Synchronization  Landscape  shows  that  the  predominant 
mission  awareness  integration  point  is  the  system  operator 
and  the  operator’s  display  console. 


operators 

and 

display 

consoles 


NATO  UNCLASSIFIED 


Low  q’s  in  this  view  indicate  lack  of 
mission  complexity  awareness 


~^oumezi^wExarrrina1r(^~oEa^mctDn^ModBtfnyrWsk^obwTechaTqueT^dersm7EoxertrBrownswont 
http.//www.  sei.  emu.  edu/publications/documents/06.  reports/OQsrO 1 7.  html 
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Requirements-Driven 

and 

Partnership 

(-Based 

Systems  Engineering  &  Training 

Education 


Jerrell  Stracener 
Stephen  Szygenda 
James  Rodenkirch 

October  24,  2007 
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Agenda 


•Systems  Engineering  Education 

-SMU  (Systems  Engineering  Program)  Overview 
-Program  Development 

•Systems  Engineering  Aerospace  &  Defense  Initiative 

•Systems  Engineering  Training 

•Summary 


Definitions 


•  Education 

-  “can  be  thought  of  as  the  process  of  acquiring  knowledge 
and  information,  usually  in  a  formal  manner. . .  [including] 
learning  how  to  think” 

-  “typically  measured  by  testing  comprehension  and 
knowledge  retention” 

•  Training 

-  “the  process  of  gaining  proficiency  in  some  skill  or  skill 
set” 

-  “usually  measured  by  the  learner’s  ability  to  demonstrate 
the  learned  skill  by  producing  desired  outcomes” 


Definitions  from  the  American  Society  of  Quality,  Certified  Quality  Manager  Handbook 


Agenda 


•Systems  Engineering  Education 

-SMU  (Systems  Engineering  Program)  Overview 
-Program  Development 

•Systems  Engineering  Aerospace  &  Defense  Initiative 

•Systems  Engineering  Training 

•Summary 


Objective  (of  this  section) 


To  present  the  highlights  of  a  non-traditional  university  systems 
engineering  program  that  was  initiated  and  has  been  developed 

—in  response  to  aerospace  &  defense  needs 

—with  extensive  industry  and  government  participation 

—with  DoD  OSD  /  DAU  /  Military  Services  review  &  guidance 
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SMU-EMIS 

System  Engineering  Program 


Systems  Engineering  Program  (SEP) 

Overview 
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SMU-EMIS 

System  Engineering  Program 


Mission 


•  Provide  education  relevant  to  the  engineering  of  systems 

•  Foster  and  conduct  research  in  selected  areas  of 
systems  engineering 

•  Maintain  a  Systems  Engineering  Program  in  partnership 
with  industry,  government  and  associations  that  is 
responsive  to  current  and  emerging  needs  and 
requirements 
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SMU-EMIS 

System  Engineering  Program 


Driven  by  Industry  and  Government  needs 


Engineers, 

Designers, 

Analysts 

Systems  Thinking  Skills 
Problem  definition, 
solving,  and  presentation 
skills 


Systems  Engineering 

•  Education 

•  Research 


Systems  Engineers 

•  Entry  level  skills 

•  Skills  update 

•  Career  growth 


Team  Leaders 

•  Skills  update 

•  Systems  Thinking 
Skills 

•  Systems 
Engineering  Skills 


To  help  you  become  a  better  engineer  and  manager 
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SMU-EMIS 


System  Engineering  Program 


Current  Academic  Program 
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SMU-EMIS 

System  Engineering  Program 


Academic  Programs 


•  On-Campus  and  Distance  Programs 

•MS  SE 

•  Fast  Track  Second  Masters 

•  Certificate  Series  in  SE 

•  Non-Degree  Studies  (for  credit)  in  SE 

•  PhD  in  Applied  Science  (Major  in  Systems  Engineering) 

•  On-Site  and  Virtual  On-Site  Programs 

•MS  SE 

•  Fast  Track  Second  Masters  Program 
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SMU-EMIS 

System  Engineering  Program 


MS  SE  Program  Options 


•  “Live”  on-campus  and  Distance  Students  via 
DVD 

-  Very  flexible  structure 

•  On-Site  and  Virtual  On-Site 
—  Offered  “live”  only 

—  Very  little  flexibility 
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SMU-EMIS 

System  Engineering  Program 


MS  SE  Degree  Requirements 


•  Thirty  term-credit  hours  of  graduate  courses  with  a  minimum 
GPA  of  3.00  on  a  4.00  scale. 


Satisfactory  completion  of  the  following  five  core  courses: 


EMIS  7300 
EMIS  7301 
EMIS  7303 
EMIS  7305 

EMIS  7307 


Systems  Analysis  Methods 
Systems  Engineering  Process 
Integrated  Risk  Management 
Systems  Reliability,  Supportability  & 
Availability  Analysis 
Systems  Integration  and  Test 
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SMU-EMIS 


System  Engineering  Program 


MS  SE  Degree  Requirements 


•  Satisfactory  completion  of  one  of  the  following 
tracks: 

-  Systems  Engineering  Technology  Track 

-  Systems  Engineering  and  Design  Track 

-  Logistics  and  Supply  Chain  Management  Track 

-  Systems  Engineering  Application  Track 

-  On-site  (Executive  Format)  Track 
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SMU-EMIS 

System  Engineering  Program 


MS  SE  Degree  Admission  Requirements 


•  MS  SE  Admission  Requirements 

•  Bachelor  of  Science  in  engineering*,  mathematics,  or  one  of 
the  quantitative  sciences  (*a  Bachelor  of  Science  in  an 
appropriate  engineering  discipline  is  required  for  the  Systems 
Engineering  and  Design  track) 

•  G.P.A.  of  at  least  3.00  out  of  4.00  scale  in  previous 
undergraduate  and  graduate  study. 

•  A  minimum  of  two  years  of  college-level  mathematics, 
including  at  least  one  year  of  calculus. 
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SMU-EMIS 

System  Engineering  Program 


Current  Systems  Engineering  Courses 


Course  Number 

Title 

Date  Approved 

EMIS 

7300 

Systems  Analysis  Methods 

April-2000 

EMIS 

7301 

Systems  Engineering  Process 

September- 1994 

EMIS 

7303 

Integrated  Risk  Management 

September- 1994 

EMIS 

7305 

Systems  Reliability,  Supportability  and  Availability  Analysis 

Sept  1994/Rev  Apr  2005 

EMIS 

7307 

Systems  Integration  and  Test 

September- 1994 

EMIS 

7310 

Systems  Engineering  Design 

April-2000 

EMIS 

7312 

Software  Systems  Engineering 

April-2000 

EMIS 

7315 

Systems  Architecture  Development 

April-2000 

EMIS 

7320 

Systems  Engineering  Leadership 

Apr  2000/Rev  Apr  2005 

EMIS 

7330 

Systems  Reliability  Engineering 

April-2000 

EMIS 

7335 

Human- Systems  Integration 

April-2005 

EMIS 

7340 

Logistics  Systems  Engineering 

April-2000 

EMIS 

7347 

Critical  Infrastructure  Protection/Security  Systems  Engineering 

April-2005 

EMIS 

8340 

Systems  Engineering  Software  Tools 

April-2005 

EMIS 

8342 

Six  Sigma  Systems  Engineering 

April-2005 

EMIS 

8348 

Supply  Chain  Systems  Engineering 

April-2005  _ 
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SMU-EMIS 

System  Engineering  Program 


Current  Systems  Engineering  Courses 


Course  Number 

Title 

Date  Approved 

EMIS 

7318 

Systems  Engineering  Planning  and  Management 

March-2007 

EMIS 

8305 

Systems  Life  Cost  &  Affordability  Analysis 

March-2007 

EMIS 

8307 

Systems  Test  and  Evaluation 

March-2007 

EMIS 

8310 

Collective  Systems  Design 

March-2007 

EMIS 

8315 

Innovation  Systems  Design 

March-2007 
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SMU-EMIS 

System  Engineering  Program 


In-Development  Courses 


•Introduction  to  Systems  Engineering  (Undergraduate  Course) 
•Systems  Requirements  Engineering 
•Acquisition  Logistics  Systems  Engineering 
•Sustainment  Logistics  Systems  Engineering 
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SMU-EMIS 

System  Engineering  Program 


Systems  Engineering  Course  Development  Process 
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SMU-EMIS 

System  Engineering  Program 


SMU  Systems  Engineering  Research  Focus 


Program  SE  and 
Concept  Development  SE 
For  Complex  Systems 


SoN/Rqmts 

Concept 

SDD  // 

Development 

Development 

>> 

Rqmts  Analysis 
and  Refinement 

Production 


Sustainment  SE 
For  Complex  Systems 


Executive  Review  10.10.07 
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SMU-EMIS 

System  Engineering  Program 


Funded  Research 


Funded  by 

Title 

Dates 

Amount 

U.S.  DoD  DAU  / 
NAVYSPAWAR 

System  Engineering  in  Science  and 
Technology 

October  1,  2007  -  September  31,  2008 

$40,000 

Lockheed  Martin 
Aeronautics  Co. 

Development  of  Response  Framework  to 
Regional  Systems  Engineering  Education, 
Research  and  Training  Needs 

August  10,  2007  December  15,  2007 

$40,000 

U.S.  Army-  ISEC 

Phase  I:  Re-engineering  Not-for-profit 
Technical  Organizations  for  Transition  to 
Market- Driven  Enterprises:  Strategies, 
Models,  and  Application  to  the  Technical 
Information  Center 

September  15,  2005-  September  20,  2006 

$89,488 

Lockheed  Martin 
Missiles  &  Fire 
Control 

Potential  Capability  Maturity  Model, 
Integrated™  (CMMI)  Generic,  Practice 
(GP)  and  Specific  Practice  (SP)  Tailoring 
Approaches 

September  15,  2005-  May  31,  2006 

$50,000 

U.  S.  Navy- 
SPAWAR 

Phase  II:  CIP  Systems  Engineering  for 
Critical  Infrastructure  Protection  Center 
(CIPC) 

November  16,  2004-February  28,  2005 

$60,000 

U.  S.  Navy- 
SPAWAR 

Phase  I:  CIP  Systems  Engineering 

April  13,  2004  -September  30,  2004 

$60,000 
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SMU-EMIS 

System  Engineering  Program 


PhD  AS  (SE)  Student  Focus 


•  Target  Students 

-  Primary  -  Full-time  Aerospace/Defense  Sector  employees; 
industry  and  government 

-  Secondary  -  Full-time  students  funded  by  government  and 
industry  research  grants 

•  Target  Students  Profile 

-  Engineering  and  other  Technical  degrees 

-  Work  experience  in  Aerospace/Defense  sector 

-  U.S.  Citizens  with  active  DoD  Security  Clearances 
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SMU-EMIS 

System  Engineering  Program 


Resident  SE  Faculty 


•  Jerrell  Stracener,  Ph.D. 

•  Steve  Szygenda,  Ph.D.* 

•  Junfang  Yu,  Ph.D.  * 

•  Eli  Olinick,  Ph.D.* 

•  Mitch  Thornton,  Ph.D.* 


Scholar  in  Residence  &  Founding  Director 

(Vought/Northrop  Grumman) 


Professor,  Cecil  H.  Green  Chair 
(AT&T  Bell  Labs) 

Assistant  Professor 

(12) 

Associate  Professor 


Professor 

(E- Systems  Greenville) 


*=  Part  time  SE  Program 
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SMU  -  EMIS 

System  Engineering  Program 


SE  Adjunct  Faculty 


•  Arunski,  Karl  P.E. 

•  Bell,  Bob 

•  Bell,  Dave  ,  DE 

•  Broihier,  Ann 

•  Chollar,  Jr.  George  ,  PhD 

•  Cluff,  Kevin  PhD,  P.E. 

•  Cowin,  Howard 

•  Daley,  Gunter 

•  Delzer,  Dennis,  PhD 

•  Durchholz,  Matt,  PhD 

•  Hinderer,  Jim,  PhD 

•  Hopper,  Mike,  DE 

•  Ibarra,  Gerard,  PhD 

•  Lipp,  John,  PhD 

Executive  Review_1 0.1 0.07 


Raytheon  Intelligence  and  Info.  Sys. 

Lockheed  Martin  Aeronautics  Company 
Mitre 

Raytheon  Network  Centric  Systems 
Statistical  Design  Institute,  LLC 
Abbott  Laboratories 

Lockheed  Martin  Missiles  &  Fire  Control 
Siemens  Government  Services 
Raytheon  Space  and  Airborne  Systems 
Lockheed  Martin  Missiles  &  Fire  Control 
Raytheon  Space  and  Airborne  Systems 
L-3  Communications  Integrated  Systems 
Ibarra  &  Associates 

Lockheed  Martin  Missiles  &  Fire  Contco^^K 
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SMU-EMIS 

System  Engineering  Program 


SE  Adjunct  Faculty 


continued 


•  Lyons,  Jan,  PhD 

•  Muto,  William,  PhD 

•  Oshana,  Rob 

•  Rynas,  Chris 

•  Sampson,  Mark 

•  Skinner,  Steve 

•  Vacante,  Russell 


Lockheed  Martin  Missiles  &  Fire  Control  (Ret.) 

GE  Medical  Systems 

Freescale  Semi-conductor 

Raytheon  Space  and  Airborne  Systems 

Siemens  Automation 

Bell  Helicopter 

US  DoD  defense  Acquisition  University 


Executive  Review_1 0.1 0.07 


24 


SMU-EMIS 

System  Engineering  Program 


SMU  System  Engineering  Program 

MS  SE  Admissions  and  Graduates 


Phase  150 


100 


II 


III 


IV 


50 


■  Admissions 


□  Graduated 


Forecast  = 


1995 


11 


J] 


1996  1997  1998 


10 


14 


21 


30 


36 


22 


13 


10 


1999:2000  2001  2002  2003  2004  2005 


72  142 


2006  ■'  2007  2008 


100  81 


91 


63 


2009 


Cumulative  admitted  as  of  September  14,  2007:  73 1 
Cumulative  graduated  as  of  August  15,  2007:  411 

Note:  Does  not  include  NTU  Students 
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SMU-EMIS 

System  Engineering  Program 


SEP  Development 


26 


SMU-EMIS 

System  Engineering  Program 


Development  Model 

Industry-  Government  -  Student  Partnership 


^O' 
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SEP 


Industry  & 
Government 
Organizations 


needs 
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SEP  Development 
Team 


Development 

Projects 


Education 


Research 


Students, 

Engineers 

Managers 

- i -  - 1 - 

needs 

_ i  ■ 

l . 

Review  & 

Evaluation 
- 1 - 
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At'\cS  ,,  .WAV'40" 
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SMU-EMIS 

System  Engineering  Program 


Development  Highlights 


•  Initiated  Feasibility  Study . September  1991 

•  Established  ad  hoc  SE  Advisory  Council . January  1992 

•  Initiated  Proposal . February  1992 

•  Investigated  Launching  SEP  at . April  1993 

-  UT  Arlington 

-  UT  Dallas 

-  SMU 


(estimated  400  to  500  admissions  in  first  10  years) 

•  Selected  SMU . June  1993 

•  Delivered  Proposal  to  SMU  SoE . July  1993 

•  SMU  Board  of  Trustees  Approved  Proposal . December  1994 

•  MS  SE  Degree  Program  Launched . Jan. 

-  SMU  -  EMIS 

Executive  Review_1 0.1 0.07 

System  Engineering  Program 


ad  hoc  Systems  Engineering  Advisory  Council 

1991-1995 


Name 

Organization 

Location 

Arunski,  Karl,  P.E.** 

Texas  Instruments,  Inc. 

Dallas,  TX 

Coyne,  Bill 

American  Airlines 

Fort  Worth,  TX 

Davis,  Joe,  P.E. 

Loral  Vought  Systems 

Grand  Prairie,  TX 

Dean,  Joe,  Ph.D. 

Lockheed  Martin  Tactical  Aircraft  Systems 

Fort  Worth,  TX 

Halligan,  Charles 

General  Electric  Transportation  Systems 

Erie,  PA 

Hanson,  Harold 

EDS 

Plano,  TX 

Harris,  Doug,  DE 

Southern  Methodist  University 

Dallas,  TX 

Jain,  Anant,  Ph.D. 

Rockwell  International 

Richardson,  TX 

Kolson,  Joanna 

Federal  Reserve  Bank 

Dallas,  TX 

Luhks,  Ronald,  Ph.D. 

Loral  Aerospace 

Houston,  TX 

Martin,  Kim 

Abbott  Labs 

Irving,  TX 

Pearse,  Derek 

Hughes  Training,  Inc. 

Arlington,  TX 

Ransom,  C.  J. ,  Ph.D. 

Bell  Helicopter  Textron 

Arlington,  TX 

Stracener,  Jerrell,  Ph.D.* 

Vought/Northrop  Grumman  Corp. 

Grand  Prairie,  TX 

Shaw,  Terry,  Ph.D. 

E- Systems 

Greenville,  TX 

Steinheimer,  Steven  L. 

E- Systems 

Garland,  TX 

Tucker,  Scott 

Hughes  Training,  Inc. 

Arlington,  TX 

Vacante,  Russell,  Ph.D. 

Army  Management  Staff  College 

Fort  Belvoir,  VA 

Zsak,  Mike 

U.S.  DoD  OSD 

Washington,  DC 

*=Chairman 
**=Vice  Chairman 
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SMU  -  EMIS 

System  Engineering  Program 


SEP  Business  Structure 


•  Multidisciplinary  Program  -  Department  Independent 

•  Build  on  Aerospace  &  Defense  (A&D)  Base  &  Needs 

•  Focus  on  part-time  students  employed  full-time  by  the  A&D 
sector  -  Industry  &  Government 

•  Utilize  SE  subject  matter  experts  employed  by  A&D  for 
Adjunct  Faculty  for  teaching  most  courses  -  Scalable 

•  Grow  number  of  resident  faculty  to  develop  SE  research  & 
PhD  SE  programs  and  teach  specialized  advanced  SE  courses 


ad  hoc  SE  Council  recommendations 
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SMU-EMIS 

System  Engineering  Program 


Development  Highlights 


•  Phase  I  -  Concept  Exploration  &  Proposal  (Sept  1 99 1  -  Dec  1 994) 

•  Phase  II  -  Start-Up  and  Development  (Jan  1 995  -  Dec  1 999) 

•  Phase  III  -  Rqmts.  Driven  Development  (Jan  2000  -  Dec  2006) 

•  Phase  IV  -  Focused  Development  (Jan  2007  -  Dec  2011) 
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SMU  -  EMIS 

System  Engineering  Program 


Executive  Reviews 


•  U.S.  DoD  Defense  Acquisition  University 

-  Dr.  Russell  Vacante,  Director 

•  Lockheed  Martin  Aeronautics 

-  Tom  Blakely,  VP  Engineering 

-  Bob  Manny,  VP  Enterprise  Integration 

-  Jim  Engelland,  VP  Systems  Engineering  &  Chief  Engineer  F-35 

-  Frank  Cappuccio,  VP  Advanced  Development  Programs 

-  Bill  Anderson,  VP  Engineering 

•  Lockheed  Martin  Missies  &  Fire  Control 

-  Glenn  Miller,  VP  Technical  Operations 

-  Bill  Cannon,  VP  Engineering 

•  Raytheon  Information  and  Intellegence  Systems 

-  John  Grimm,  VP  Engineering 
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SMU  -  EMIS 


System  Engineering  Program 


Executive  Reviews 


•  U.S.  DoD  OSD 

-  Bob  Skalamera,  Deputy  Director,  Systems  and  Software  Engineering 

-  Mark  Schaeffer,  Deputy  Director,  Defense  Systems,  and  Director,  Systems  Engineering, 
OUSD  (AT&L) 

-  Dr.  James  Roche,  Secretary  of  the  Air  Force 

-  Mike  Zsak,  Director,  Systems  Engineering 

-  Mike  McGrath,  Director  CALS 

•  Raytheon  Space  &  Airborne  Systems 

-  Bob  Kern,  VP  Engineering 

-  Janne  Ackerman,  Director  North  Texas  Engineering 

-  Bob  Rassa,  Director  Systems  Supportability 

•  Vought  Aircraft  Company 

-  Eric  Smith,  Senior  VP  Programs 

-  Joe  Ayers,  VP  Engineering 


L-3  Communications  Integrated  Systems 
-  Dr.  Val  Gavito,  VP  Engineering  &  Strategic  Initiatives 


Executive  Review_1 0.1 0.07 
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SMU  -  EMIS 

System  Engineering  Program 


DAU  -  SMU  SEP  Partnerships 


•  U.S.  DoD  Defense  Acquisition  University  (DAU)  and  SMU 
Systems  Engineering  Program  (SEP)  MoU 

1.  Provide  members  of  U.S.  DoD  Acquisition,  Technology, 
and  Logistics  (AT&L)  workforce  the  opportunity  to  apply 
courses  provided  by  DAU  towards  a  SMU  graduate  degree 
in  systems  engineering. 

2.  Provide  SMU  SEP  students  access  to  DAU  courses,  and 

3.  Collaboratively  develop  research  topics  and  projects  in 
systems  engineering. 


•  U.S.  Navy  SPAWAR  Charleston  and  SMU  SEP  Cooperative 
Research  and  Development  Agreement  (CRADA)  in  work 
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SMU-EMIS 

System  Engineering  Program 


National  Defense  Industrial  Association 

SYSTEMS  ENGINEERING  DIVISION 


* 


STEERING  COMMITTEE 

Carlos  Galvan  -  SPC 

Dr.  Ronald  Johnson  -  Boeing 

Pat  Bevins  -  GD,  Electric  Boat 

Hank  Eyster  -  Harris  Corp 

Bill  Chen  -  United  Defense 

Nick  Fritz  -  Burdeshaw  Associates 

Jeff  Dutton  -  Sverdrup 

Dr.  Rob’t.  Lentz  -  General  Dynamics 

Jim  Sturges  -  Lockheed  Martin 

open-  ManTech  Inti 

Richard  Rothman  -  SA  1C 

Brooks  Nolan  -  L3  COM 

Aaron  Fuller  -  CSC 

Affiliations  &  Liaisons 

Stan  Siegel  &  Ed  Viau-  AIA 

Ken  Ptack  -  INCOSE 

Jerrell  Stracener  -  SMU 

Steve  Kuehl  -  AIAA 

Paul  Croll  -  IEEE  Computer  Soc. 

Mike  Cardinale  -  IEEE  AESS 

Rene  Smith  -  SOLE 

Les  Orlidge  -  IEEE  SCC20 

Elliot  Axelband  -  RAND  Corp 

Greg  DiBenedetto  -  GEIA 

Open  -  Mitre 

Brian  Gallagher  -  SEI 


CHAIR 

Bob  Rassa 

Raytheon 

(IEEE) 

VICE-CHAIR 

Hal  Wilson 

Northrop  Grumman 
(EIA) 


NDIA 

COMMITTEE  EXECUTIVE 

Sam  Campagna 


CMMI  Project 
Co-Chairs 

Mike  Nicol,  USAF 

Bob  Rassa,  Raytheon 

Interoperability 

Committee 

Tom  Croak, CSC 
Jack  Zavin,  OASD(NII) 


Automatic  Test 
Committee 

DaveWallhermfechtel,  Boeing 
Les  Orlidge,  AAI  Corp 
Joe  O’Connell,  Boeing 


GOVT  Steering  Group 
Mark  Schaeffer  -  OSD  (Chair) 
Dr.  V.  Garber  -  OSD(AT&L)DS 
RADM  Kate  Paige  -  MDA 
Mark  Wilson  -  USAF 
Jesse  McCurdy  -  USN 
Lou  Kratz  -  OSD 
Robert  W.  Schmitt-  DCMA 
Art  Pyster  -  FAA 
Open  -  NASA 

Bob  Skalamera  -  OSD(AT&L) 
John  Osterholz  -  OSD(C3l) 
RADM  Mike  Mathis  -  USN/JS 
Open  -  Army 
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SMU-EMIS 

System  Engineering  Program 


Plans 
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SMU-EMIS 

System  Engineering  Program 


Baseline  Academic  Programs 


Entry  Point 


t 


Systems  Engineering  Education  is  a  Journey  -  Not  a  Destination 


SE  Research  ProjectRegional  Response_09.24.07 
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SMU-EMIS 

System  Engineering  Program 


Summary 
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SMU-EMIS 

System  Engineering  Program 


Summary 


•  A  SE  Education  &  Research  program  with  focus  on: 

-  aerospace  &  defense 

-  development  of  complex  Systems  (as  opposed  to  acquisition) 

•  Track  record  of  success  in  responding  to  Customer  needs 

-  SEP  Established  in  1994 

-  Growing  Enrollment  and  expanding  scope 

-  Extensive  &  growing  industry  and  government  network 

•  SE  is  currently  a  HOT  topic  (but  lacks  branding) 

-  Emphasis  on  SE  by  U.S.  DoD  and  defense  contractors 

-  High  and  increasing  Student  interest  in  SE  ( not  in  becoming  a  SE,  but  rather  in 
utilizing  SE  education  to  become  a  “letter”  engineer  or  for  career  advancement) 

-  Increasing  number  of  University  SE  Programs  (but  many  are  commingled  with 
other  programs) 

•  The  SEP  is  severally  resource  constrained  for  PhD  SE 
generation  and  research 


Executive  Review_1 0.1 0.07 
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SMU  -  EMIS 

System  Engineering  Program 


Agenda 


•Systems  Engineering  Education 

-SMU  (Systems  Engineering  Program)  Overview 
-Program  Development 

•Systems  Engineering  Aerospace  &  Defense  Initiative 

•Systems  Engineering  Training 

•Summary 
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PRIVATE 


Description  — 

•  Research  based  exploration  and  definition  of  a 
framework  for  effective  response  to  regional 
industry  and  government  systems  engineering- 
education, -research  and-training  &  consulting 
needs. 

•  Initial  focus  on  the  aerospace/  defense/security 
sectors. 

•  Expansion  to  other  sectors  will  be  guided  by 
regional  needs. 


Statement  of  Work 


PRIVATE 


•  Specific  tasks  necessary  to  evolve  the  preferred 
response  framework  include  the  following: 

-  Industry  and  government  needs  captured  and  assessed 

-  Identification  and  analysis  of  regional  capabilities  and  resources,  both 
current  and  planned 

-  Analysis  to  determine  gaps  and  overlaps  with  respect  to  needs 

-  Explore  and  define  alternatives  for  responding  to  needs,  including 
benchmarking  the  nations  best. 

-  Evaluate  and  refine  alternates  to  evolve  the  preferred  concept,  a 
regional  framework. 

-  Strawman  regional  framework  development  plan 


•  To  ensure  a  structured  technical  approach  and 
balanced  solution,  the  systems  engineering  process 
will  be  utilized  in  the  planning  and  conduct  of  this 
research  project. 
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Fannin 


Texarkana 


TAMU 


El  Paso 
•  UTEP 


Vought 


Longview 

•  LeTourneau 

Tyler 

•  UT  Tyler 

Houston 

•  UHCLC 


San  Antonio 
•  UTSA 


Regional 


* 
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SMU 


Navarro 


LMMFC 


Leveraging  Regional  Capabilities  to  Meet  Regional  Needs 
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K^T7?7^T7?!^vT»7#/ii 

IfeiMNH'  f  7 

U  Ir  '  V  1 

??•;{  fii 

f  1 

1  J\ 

■  V  /  v 

■tf 

^ 1 

D/FW 

•SMU 

UNT 

UTD 

•UTA 

TCU 

TAMU 


Lockht 
Raythei 
Boeing 
L-3  Cor 
Bell 
EDS 
Vought 
Etc. 


Mobilize  Resources  and  Build  on  Experience 


PRIVATE 


•  Initiate  SE  Tiger  Team  of  members  Industry, 
Government  and  University  Affiliations 


•  Utilize  Previous  Start-Ups  as  Guides 

-  SAE  RMSL  Division  (G-l  1):  1985  -  2000 

-  CALS  Connectivity  Center  (CCC)  /  UTA  ARRI: 
1989  -  1999 

-  SMU  Systems  Engineering  Program:  1991  - 
Present 


Organization 


*=  to  be  invited 


organizational  affiliation  -  not  representation 


SE  Research  Project_Regional  Responsel  0.08.07 
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Research  Process 


PRIVATE 


August  10,  2007 


December  7,  2007 


Vision 


PRIVATE 


A  National  Center 
With  Regional  Focus 


SE  Research  Project  Regional  Response_09.28.07 
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Center  for  SE  -  Functional  Concept  -  Overview 


PRIVATE 


Agenda 


•Systems  Engineering  Education 

-SMU  (Systems  Engineering  Program)  Overview 
-Program  Development 

•Systems  Engineering  Aerospace  &  Defense  Initiative 

•Systems  Engineering  Training 

•Summary 
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Training  Goal 


to  provide  System  Engineering  training  that  is 

-  Tailored  to  customer  needs  and  work  place 

-  Delivered  by  industry,  government  and  academia  subject 
matter  experts 

-  Relevant 

-  Conducted  in  an  interactive  workshop  format 
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Systems  Engineering,  LLC 


Training  Objective 

•  to  increase  Systems  Engineering  awareness 

•  to  increase  organizations  Systems  Engineering 
capability 

•  to  increase  individual  engineers  SE  expertise 
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Systems  Engineering,  LLC 


Training  Scope  &  Delivery 


System  Engineering  training  Scope 

-  Integrated  program 

-  Stand-alone  modules 

-  Special  aligned  systems  engineering  topics 

•  Delivery 

-  JGR  Systems  Engineering,  LLC 
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Systems  Engineering,  LLC 


Leveraged  and  Work-Place  Relevant  Training 


A 
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\GR  Systems  Engineering,  LLC| 


Systems  Engineering,  LLC 


Organizational  Level,  Training  “Depth”  and  Value 


Division  or  Department  level 

-  Strategic  or  tactical  business  importance 

-  Overall  value  to  a  group,  division,  or  department. 


@  the  Immediate  Supervisor,  Branch 
head  or  Project  Lead  level 

-  Impact  on  projects/programs 

-  Impact  on  employee  competence 


@  the  individual  engineer’s  level 

-  Impact  on  the  individual’s  specific  competency;  e.g.,  ability,  capabilities,  skills 

-  Value  of  those  competencies  to  the  company 
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Systems  Engineering,  LLC 


Systems  Engineering  Courses 


Customer  Tailored  training  from  1-5  days,  in  increments  of  one  day 

•  Systems  Analysis  Methods 

•  Systems  Engineering  Process 

•  Integrated  Risk  Management 

•  Systems  Reliability,  Supportability  and  Availability  Analysis 

•  Systems  Integration  and  Test 

•  Systems  Engineering  Design 

•  Software  Systems  Engineering 

•  Systems  Architecture  Development 

•  Systems  Engineering  Leadership 

•  Systems  Reliability  Engineering 

•  Human- Systems  Integration 

•  Logistics  Systems  Engineering 
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Systems  Engineering,  LLC 


Systems  Engineering  Courses 


•  Critical  Infrastructure  Protection/Security  Systems  Engineering 

•  Systems  Engineering  Software  Tools 

•  Six  Sigma  Systems  Engineering 

•  Supply  Chain  Systems  Engineering 

•  Systems  Test  and  Evaluation 

•  Systems  Engineering  Planning  and  Management 

•  Systems  Cost  Engineering 

•  Systems  Life  Cycle  Logistics 

•  Innovative  Systems  Design 

•  Systems  Modeling  and  Simulation 

•  Systems  Prognostic  and  Health  Management 

•  Systems  Development  Program  Engineering  and  Management 
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Systems  Engineering,  LLC 


Summary 


•  Work-place  Relevant  Systems  Engineering  Training 

-  By  Subject  Matter  Practioners 

-  Tailored  to  Customer  Needs 

•  Linkage  to  Graduate  Systems  Engineering  Courses 

•  Aerospace  &  Defense  focused 
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Systems  Engineering,  LLC 


Agenda 


•Systems  Engineering  Education 

-SMU  (Systems  Engineering  Program)  Overview 
-Program  Development 

•Systems  Engineering  Aerospace  &  Defense  Initiative 

•Systems  Engineering  Training 

•Summary 


59 


NDIA  10th  Annual  Systems  Engineering 

Conference 


“Discussion  of  the  US  Army  RDECOM 
APS  Objective  Trade  Study” 

October ,  2007 


Frank  Salvatore 

High  Performance  Technologies,  inc. 

3159  Schrader  Road 

Dover  NJ,  07801 

(973)  442-6436  ext  249 

fsalvatore@hpti.com 


Outline 


□Study  Description 
□Trade  Study  Process 

IPT 

Tools  Developed 

APS  Architectures 

Trade  Study  Tool  Architecture 

□Summary 


APS  Trade  Study  Description 


RDECOM  effort  led  by  the  ARDEC  System  Engineering 
Directorate 


Identify,  define,  and  evaluate  potential  Universal  (Objective) 
Active  Protection  System  (APS)  approaches  for  the  Future 
Combat  System  (FCS). 


Provide  decision  makers  the  tools/data  to  help  identify 
RDECOM’s  Science  and  Technology  investments  needed  to 
get  to  an  objective  APS  system. 


Trade  Study  Process 


Trade  Study  Based  on  Disciplined  &  Structured  Process 


Used  an  IPT  approach 


ARDEC 


Idaho  National  Laboratory 


Excellence  in  Analysis 


The  Trade  Study  was  a  Team  Effort 


1.0-2.0-3.0  -5.0  Requirements  -  Goals  -  Criteria  - 
HBSSSH  \  Weights  &  Utility 


Requirements  and  Stakeholders  Drive  Decision  Criteria 


4.0  Collect  Component  Database  on 

Criteria 


□  Technologies  list  build  based  on  surveying 
R&D  community  thru  several  technical 
interchange  meetings. 

Technology  specific  performance  characteristics 
established 

Data  call  to  Industry  and  Government 


□  Series  of  Data  Validation  meetings  to  confirm 
data  used  in  study  was  accepted  by 
community. 

Performance  Values 
TRL 


This  took  a  lot  of  coordination  and  cooperation 
between  Government  and  Industry  to  get  right!!!! 


6.0  Define  Uncertainty  Factors 


□  Data  Uncertainty  assessed 
by  determining: 

Component  TRL 
Data  Confidence 


□  Data  Uncertainty  applied  to 
criteria  scores  to  determine 
plus  and  minus  range 


Data  uncertainty  helped  visualize  Results  and  risk!!! 


7.0  Identify  &  Define  Alternatives 


•  List  Systems/Components 

•  Previous  Trades  •  Existing  Systems 

•  Component  Data  •  Analysis  Method, Tools  •  System  Alternatives 

•  Requirements  •  System  Assumptions  •  System  ID 


•  Integrate  System  Candidates  ‘Analyze  System  Candidate  Potential 

•  Organize  Component  Data  •  Timeline 

•  ID  Functional  Architectures  •  Accuracy 

•  Component  Compatibility 


System  and  Technology  Architectures  Required!!!!! 


7. 1  Candidate  Systems 


All  Technology  Combinations  Were  Evaluated 


Function  Definitions  (1  of  2) 


Function 

Definition 

Detect,  Acquire 

Measure  and  report  an  event  not  due  to  ambient  noise 

Declare 

Measure  and  report  an  persistent  object  that  should  be  tracked 

Classify 

Measure  and  report  what  the  persistent  object  is  either  by  class  or  specific 
type/item. 

Coarse  Track 

Measure  and  report  an  object  and  determine  that  it’s  trajectory  point  of  closest 
approach  to  our  platform  is  threatening.  Classify  and  coarse  track  may  be 
based  on  the  same  measured  data  set  and  completed  at  the  same  time 

Initial  Slew 

Initial  slew  of  launcher  to  launch  position  using  fire  control  solution  based  on 
coarse  track 

Initial  Tube  Selection 

Initial  designation  of  launch  tube  or  tubes  in  fixed  system  that  need  to  be 
“warmed  up”  using  fire  control  solution  based  on  coarse  track 

Fine  Track 

Measure  and  report  a  target  to  enable  calculation  of  a  fire  control  solution 

Fine  Slew  &  Fire  Control 

Slew  launcher  to  final  position  and  launch  an  interceptor  loaded  with  any 
required  flight  path,  terminal  guidance,  and  fuzing  information 

Final  Tube  Selection  &  Fire 
Control 

Final  designation  of  launch  tube  in  fixed  system  and  launch  an  interceptor 
loaded  with  any  required  flight  path,  terminal  guidance,  and  fuzing  information 

APS  system  functions  defined  from  all  technology  components  and  systems  studied. 


Function  Definitions  (2  of  2) 


Function 

Definition 

In-Flight  Track 

Measure  and  report  a  target  trajectory  to  provide  in-flight  guidance  to  an 
interceptor 

No-Op 

“No  operation”  -  used  to  designate  function  not  performed 

In-Flight  Guidance 

Propulsion  to  change  flight  path  of  interceptor 

Terminal  Track 

Measure  and  report  a  target  trajectory  to  provide  terminal  guidance  &  fuzing 
updates  to  an  interceptor 

Terminal  Guidance  &  Fuze 

Orient  (focus)  the  warhead  to  produce  the  desired  effect  &  initiate  the  effect  at 
the  prescribed  time  and  /  or  the  prescribed  distance  from  target 

Warhead  Effect 

Target  negation 

APS  system  functions  defined  from  all  technology  components  and  systems  studied. 


Generic  APS  Architectures 


Architectures  for  Unguided  Interceptors 

Architectures  for  Guided  Interceptors 

U1 

U2 

U3 

U4 

G1 

G2 

G3 

G4 

System 

Functions 

Detect,  Acquire  &  Declare 

Passive  Cuer 

Passive  Cuer 
/  Coarse 
Tracker 

Passive  Cuer 

Active  Cuer  / 
Tracker 

Passive  Cuer 

Passive  Cuer 
/  Coarse 
Tracker 

Passive  Cuer 

Active  Cuer  / 
Tracker 

Classify 

Active  Tracker 

Passive  or 
Active  Coarse 
Tracker 

Active  Tracker 

Passive  or 
Active  Coarse 
Tracker 

Coarse  Track 

Initial  Slew  /  Tube 

Selection 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Fine  Track 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

Final  Slew  /  Tube  Selection 
&  Fire  Control 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

In-Flight  Track 

None 

None 

None 

None 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

In-Flight  Guidance 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Terminal  Track 

Unguided 

Interceptor 

Unguided 

Interceptor 

Unguided 

Interceptor 

Unguided 

Interceptor 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

Terminal  Guidance  &  Fuze 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Warhead  Effect 

Functional  allocation  to  components  provided  context  for  data  provided  on  specific 
components  and  was  critical  in  both  the  Timeline  and  Accuracy  Analysis. 


Architecture  U1 


Architecture  U2 


Architecture  U3 


Architecture  U4 


Passive  Cuer 
Detect,  Acquire  &  Declare 


Classify 


i 


Coarse  Track 

r 

Fine  Track 

r 

In-Flight  Track  ■ 

r 

Terminal  Track 

Active  Tracker 


Architecture  G1 


Launcher 


Initial  Slew  /  Tube  Selection 


Fine  Slew  /  Tube 
Selection  &  Fire  Control 


In-Flight 

Guidance 


Terminal  Guidance  &  Fuze 


Warhead 

Effect 


Guided  Interceptor 


Passive  Cuer  /  Coarse  Tracker 
Detect,  Acquire  &  Declare 


▼ 

Classify 


i 


Coarse  Track 

r 

Fine  Track 

r 

In-Flight  Track 

r 

Terminal  Track 

Active  Fine  Tracker 


Architecture  G2 


Launcher 


Initial  Slew  /  Tube  Selection 


Fine  Slew  /  Tube 
Selection  &  Fire  Control 


In-Flight 

Guidance 


Terminal  Guidance  &  Fuze 


Warhead 

Effect 


Guided  Interceptor 


Architecture  G3 


Detect,  Acquire  &  Declare 


Classify 


i 


Coarse  Track 

1 

r 

Fine  Track 

r 

In-Flight  Track 

1 

r 

Terminal  Track 

Active  Cuer  /  Tracker 


Architecture  G4 


Launcher 


Initial  Slew  /  Tube  Selection 


Fine  Slew  /  Tube 
Selection  &  Fire  Control 


In-Flight 

Guidance 


Terminal  Guidance  &  Fuze 


Warhead 

Effect 


Guided  Interceptor 


7.2  Evaluate  Candidates 


7.1  Candidate  Systems 


Compatibility 

Analysis 

7.2.3 


Timeline  Analysis[ 
7.2.1 


N 


Command  Guided  Blast  with  II 


RF  Homing  Guided  Blast 


Command  Guided  Focussed  F  >g  with  IR  sensor 


g  Guided  Focused  f  ig  w/  IR  Sensor 


“  Y 

1 

r 

—I 

I 

Accuracy  Analysis 
7.2.2 

Requirements 


System  ID 

Cuer 

Tracker 

Launcher 

Interceptor 

X-4986 

Cue  Component  ID 

Track  ID 

Launcher  ID 

Interceptor  ID 

Y 


7.3  Define  Alternatives 


System  ID  was  key  to  configuration  control  and 
essential  to  manage  resulting  data. 


APS  Trade  Study  Tool  Architecture 


Abstract  Architecture 

□  Schematic  Block  Diagrams 
Physical  Architecture 
Interfaces 


Formal  Architecture 

□  IDEFO,  FFBD,  EFFBD,  Hierarchy 
Physical  Architecture 
Functional  Architecture 
Interfaces 
Data  Flow 


[Threat  Community!  [S&T  Community] 
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Report 

7.2.1  Timeline  Analysis 
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Timeline  Analysis  was  a  first  order  filter  used  to  Identify  Technology  Combinations  that  do  not 
have  potential  to  achieve  FCS  Objective  APS  requirements. 


7.2.3  Compatibility  Analysis 
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-  Electrical  -  signals,  voltage,  power 

Software  interface  considerations  include  added  requirements  for 
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-  Data  processing  and  algorithms 
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Compatibility  Analysis  was  used  to  determine  if  the  Technology  Combinations  interfaces  were 
compatible  and  could  realistically  be  combined  to  form  a  system. 


7.3  Define  Alternatives 
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8.0  Evaluate  /  Score  Alternatives 


Capture  Design  Details  of  each  system  that  passes  the  timeline  and  accuracy  analysis. 


7.0  Identify  &  Define  Alternatives 


•  List  Systems/Components 

•  Previous  Trades  •  Existing  Systems 

•  Component  Data  •  Analysis  Method, Tools  •  System  Alternatives 

•  Requirements  •  System  Assumptions  •  System  ID 


•  Integrate  System  Candidates  ‘Analyze  System  Candidate  Potential 

•  Organize  Component  Data  •  Timeline 

•  ID  Functional  Architectures  •  Accuracy 

•  Component  Compatibility 


System  and  Technology  Architectures  Required!!!!! 


8.0  Score  APS(s)  Alternatives 


Measure  how  well  each  of  the 
systems  meets  the  Goals! 
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9.0  Performance  Values/Utility 
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Summary 


□  Using  the  program  requirements  to  derive  the  evaluation  criteria  made  the  trade 
study  results  traceable  to  user  needs. 

□  Involving  all  stakeholders  early  and  often  allowed  for  acceptable  end  results. 

□  Establishing  a  System  ID  scheme  was  key  to  configuration  control  and  essential 
to  manage  resulting  data. 

□  Capturing  System  Architectures  was  essential  to  understand  how  to  model 
system  time  function  and  communicate  it  to  the  community. 

□  Tool  Architecture  helped  to  communicate  how  each  tool  was  used  in  the  trade 
study  process. 

□  As  a  result  of  capturing  the  tool  architecture 

many  tool  interface  gaps  were  identified  and  fixed. 

The  Schematic  Block  diagram  was  updated  to  be  more  correct. 

□  Tool  Architecture  was  valuable  to  communicate  with  each  tool  developer 
interfaces 

□  Modeling  and  Simulation  was  a  key  player  in  conducting  the  APS  Trade  Study 
and  helped  to  drive  decisions.  This  study  could  not  be  don’t  without  using 
models. 

□  Using  a  defined  process  were  all  stakeholders  were  involved  and  had  a  voice 
yielded  results  the  community  could  accept. 


The  Systems  Engineering  Process  was  instrumental  to  the  success  of  the  APS 

Trade  Study. 
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OTS  Tool  with  the  external  system  functions  with  which  it  interfaces 
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The  Hierarchy  Diagram  was  a  quick  way  to  quickly  capture  all  the  Trade  Study  Tools 
and  their  Hierarchical  relationships.  These  ultimately  became  the  configuration  items 

that  were  kept  under  version  control. 
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The  functional  hierarchy  diagram  emerged  from  the  architecting  process  as  a  functional 

decomposition  of  the  trade  study  analysis  effort. 
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The  IDEFO  diagram  of  the  APS  Tool  shows  both  external 
and  internal  information  interactions  between  functions 
and  the  components  performing  functions 


Perform  APS  Analysis  FFBD 


The  FFBD  (Function  Flow  Block  Diagram)  of  the  APS  Tool  shows 
the  sequencing  and  control  flow  of  the  functions  of  the  Tool 
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The  IDEFO  proved  to  be  a  rigorous  analysis  of  each  tools  inputs  and  outputs.  The 
process  of  building  this  diagram  resulting  in  discovering  several  tool  interface 

issues  that  we  had  to  go  back  and  fix. 
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The  EFFBD  (Enhanced  Function  Flow  Block  Diagram)  of  the  APS 
Tool  shows  both  the  data  flow  and  control  flow  of  the  Tool 
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Summary 


□  Using  the  program  requirements  to  derive  the  evaluation  criteria  made  the  trade 
study  results  tradeable  to  user  needs. 

□  Involving  all  stakeholders  early  and  often  allowed  for  acceptable  end  results. 

□  Establishing  a  System  ID  scheme  was  key  to  configuration  control  and  essential 
to  manage  resulting  data. 

□  Capturing  System  Architectures  was  essential  to  understand  how  to  model 
system  time  function  and  communicate  it  to  the  community. 

□  Tool  Architecture  helped  to  communicate  how  each  tool  was  used  in  the  trade 
study  process. 

□  As  a  result  of  capturing  the  tool  architecture 

many  tool  interface  gaps  were  identified  and  fixed. 

The  Schematic  Block  diagram  was  updated  to  be  more  correct. 

□  Tool  Architecture  was  valuable  to  communication  to  each  tool  developer 
interfaces 

□  Modeling  and  Simulation  was  a  key  player  in  conducting  the  APS  Trade  Study 
and  helped  to  drive  decisions.  This  study  could  not  be  don’t  without  using 
models. 

□  Using  a  defined  process  were  all  stakeholders  were  involved  and  had  a  voice 
yielded  results  the  community  could  accept. 


The  Systems  Engineering  Process  was  instrumental  to  the  success  of  the  APS 

Trade  Study. 
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Agenda 


•  Need  to  early  verify  net-centric 
information  strategies 

•  Mission  Level  Model  (MLM) 
experimentation  for  net  centric  C2 


2 


Net  Centric  Operations 

•  An  information  superiority-enabled  concept  of  operations  that 
generates  increased  combat  power  by  networking: 

—  Sensors 
—  Decisionmakers 
—  Shooters 

•  Achieve: 

—  Shared  awareness 
—  Increased  speed  of  command 
—  Higher  tempo  of  operations 
—  Greater  lethality 
—  Increased  survivability 
—  A  degree  of  self-  synchronization 

Must  define,  refine  and  early  verify  information 
strategies  that  enable  net  centric  operations 


Operationally  Effective 
Net  Centric  Information  Flows 
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DoD  needs  a  new  M&S  capability 
(MLM)  to  define,  refine  and  early 
verify  operationally  effective  net 

centric  information  flows. 


•  Net  centric  environment  facilitates 

-  Distributed  computing 

-  Distributed  storage 

—  Distributed  Command  &  Control  (C2) 

•  Net  centric  concepts  must  exploit  inherent 
concurrency  among 

—  Operations 
—  Systems 

—  Operations  and  systems 

•  DoD  is  technically  challenged  to  T&E 
complex  temporal  behavior  emerging  from 

-  Data  dependencies 

-  Control  dependencies 

-  Resource  sharing  among  activities 
—  External  asynchronous  trigger's 

•  Leading  to  difficulties  in  testing  NR  KPP 
and  its  temporal  variances  (six  sigma) 
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Need  for  Executable  Mission  Threads 


Mission  threads  have  been  the  foundation  of  DoD  acquisition 

—  Critical  Operational  Issues  are  described  via  mission  context 

—  CDD  includes  DoDAF  OV6C  to  describe  mission  threads 

—  JFCOM  is  further  refining  NECC  CDD  via  Capability  Definition  Package 
capturing  operational  threads 

—  NECC  program  is  developing  Engineering  Mission  Threads  (EMT)  for 
requirements  analysis 

—  Operational  T&E  community  describes  its  test  via  mission  threads 

Executable  mission  thread  modeling  is  a  MUST  to  develop  net  centric 
capabilities 

—  Hard  to  describe  concurrency  (implicit  in  net  centric  capabilities)  in  the  current 
textual  documentation  practice  impractical. 

—  Necessary  to  have  a  standard  to  capture  executable  mission  threads  to  compose 
mission  threads  developed  by  multiple  stakeholders  and  to  eliminate 
duplication  and  confusion 

Mission  thread  modeling  must  provide  a  collaborative  environment  to 
develop  operational  concepts  throughout  the  acquisition  cycle:  Define, 
Refine  and  Verify  capabilities 


Current  requirement  documentation  methods 
are  inadequate  for  DOD  net  centric  acquisitions 


Narrow  the  Exponential  Widening  ‘V’ 


Time  from  concept  definition  to  deployment 
increases  exponentially  with  complexity 
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MLM  101 


•  Information  exchanges  (IE)  are  information  events  among  two  entities  (systems,  operations) 

•  MLM  captures  end  to  end  information  flow  among  multiple  entities  supporting  the  mission 

—  Information  flow  is  a  sequence  of  information  events  among  mission  end  points 

•  Net  centric  operations  require  concurrent  information  flows  (mission  threads) 

—  Pipeline  allows  multiple  simultaneous  executions  of  the  same  mission  thread 
-  Parallelism  allows  simultaneous  execution  of  different  mission  threads,  which  could  share  resources 


SI  S2  S5  S7  Mission 


Selected  MLM  Technologies 

•  Based  on  standards  and  COTS  products 

•  Business  Process  Modeling  Notation  (BPMN)  OMG  standard  for  mission 
thread  modeling 

•  iGrafx  COTS  tool  for  mission  simulation  and  visualization 

•  Minitab  COTS  tool  for  design  of  experiments  and  analysis 

•  Business  Process  Executable  Language  (BPEL)  for  capturing  SOA  test 
workflow 

•  Automated  generation  of  BPEL  from  BPMN 

•  ActiveBPEL  COTS  simulation  engine  for  SOA  test 

•  SOA  standards:  SOAP,  XML  . . . 


Benefits 


•  Improves  development  &test  efficiency  via  process  automation 

•  Reduces  cost  by  implementing  automation  via  converging  standards 

•  Eases  technology  transitions  to  multiple  stakeholders  via  COTS 


MLM  Experimentation  for  C2 
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Basecase:  Experiment  #96 
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Airborne  Sensor  Case: 
Experiment  #106 


Case  #1 06  Basecase  #96 


Moving  C2  task  to  Airborne  Sensor  (AS) 


Flows 

Process  @ 

Communication  Type 

ISR  ->  TOC 

TOC 

Video/Chat 

AS->GS 

GS 

MTI/TK 

GS->JTF 

GS  and  JTF 

MTI,  Update  COP 

SOF->TOC 

SOF 

Chat 

Airborne  Sensor  Exo 

eriment  #106 

Flows 

Process  @ 

Communication  Type 

ISR  ->  TOC 

TOC 

Video/Chat 

AS-> 

AS 

Update  COP 

SOF->TOC 

SOF 

Chat 

AS:  Airborne  Sensor 
GS:  Ground  Station 

SOF:  Special  Operation  Forces  (SEAL  Team) 
TOC:  Tactical  Operations  Center 
JTF:  Headquarters/Rear 


Experimentation  Setup 

•  5  workloads 

—  1  to  5  targets 

•  360  information  Flow  Strategies 

=[6  ISR  flows]  *  [4  SOF  flows]  *  [15  AS/GS  flows] 


Sensor 

Contribution 

/Hit 

Typical  Hits 
for  F2T2 

Information 

Quality 

AS/GS 

1 

1,250 

1,250 

ISR 

25 

40 

1,000 

SIGINT 

22 

45 

990 

SOF 

65 

17 

1,105 

Fusion 

1,655 

Total 

6,000 

AS:  Airborne  Sensor 
GS:  Ground  Station 

SOF:  Special  Operation  Forces  (SEAL  Team) 
SIGINT:  Signals  Intelligence 


Improved  TST  Time  for  the  Same  Information  Quality 


52%  Improvement  in  TST  for 

processing  at  Airborne  Sensor  (AS) 
case  for  the  same  Quality  of 
Information  of  6,000  (1  Target) 


43%  Improvement  in  TST  for 

processing  at  Airborne  Sensor  (AS) 
case  for  the  same  Quality  of 
Information  of  6,000  (2  Targets) 


Performances  better  then  the  Basecase  (Red  Line,  #96) 


X-axis  =  Quality  of  Information  (HI  1 1 L^Q ua n tity ) 

Panel  variable :  Number  of  T otal  T argets 


MITRE 


Moving  processing  to  AS  has  potential  to  reduce  TST 
time  by  41%  to  52%  for  the  same  information  quality 
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Improved  Information  Quality  for  the  Same  TST  Time 


57%  Improvement  in  Quality  of 
Information  for  processing  at 
Airborne  Sensor  (AS)  case  for 
the  same  F2T2  time  of  6,575  sec 
(1  Target) 


46%  Improvement  in  Quality  of 
Information  for  processing  at 
Airborne  Sensor  (AS)  case  for 
the  same  F2T2  time  of  8,81 0  sec 
(2  Targets) 


Performances  better  then  the  Basecase  (Red  Line,  #96) 
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X-axis  =  Quality  of  Information  (IHTEL^Quantlty) 

Panel  variable :  Number  of  T otal  T argets 


Moving  processing  to  AS  has  potential  to  increase 
information  quality  by  46%-57%  for  same  TST  time 
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Operational  Capacity  for  an  Information  Strategy 

and  TST  Time 


TST  time  =  2-hours 

TST  time  =  4-hours 

AS  1 -target 

YES 

NO 

AS  2-targets 

NO 

YES 

AS  3 -Targets 

NO 

MAYBE  meet  TST 

Base  case  1 -target 

NO 

YES 

Base  case  2-targets 

NO 

NO 

•  2  Hour  TST:  Need  AS-  Information  Strategy  even  for  one  target 

•  4  Hour  TST:  AS-Strategy  can  do  2-targets  and  base  case  can  only  do  1 -target 


AS:  Airborne  Sensor  Strategy  #106 
Basecase:  Experiment  #96 
TST  =  Time  Sensitive  Targeting 
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Conclusions 

•  Acquisition  of  Net  centric  operational  capability  needs  a  new 
M&S  capability  to  support  analysis  of  required  capabilities 

—  Define,  refine  and  early  verify  mission  performances 
—  Complementary  to  net  centric  operational  exercises 

•  COTS  solutions  are  matured  enough  to  quantitatively  assess 
mission  performances  via  simulation 

—  BPMN  standard  based 

•  Further  research  is  needed  to 

—  Improve  modeling  of  the  sensor  contribution  to  commander  confidence 
—  Add  stochastic  simulation 
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Questions? 
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System  Test  and  Evaluation 
(T&E)  in  the  DARPA  Immune 
Building  Program 


Mark  Saxon 
Research  Scientist 
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CB  Attacks,  Accidents,  and  Threats 


•  Threats 

-  CWAs  and  TICS 

-  BWAs 

-  Radiological  Agents 


•  CB  Attacks  and  Accidents 

- 1984  TIC  Methyl  isocyanate,  Bhopal,  India 

-  3,800  deaths,  thousands  disabled 

- 1995  Nerve  gas  (Sarin),  Tokyo,  Japan  (subway) 

-  12  deaths,  1000+  illnesses 
-2001  BWA  Anthrax  (Florida  and  New  York) 

-  5  deaths,  10,000  treated 
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CB  Building  Protection  Overview 


Why  are  buildings  vulnerable  to  CB  attack? 

•  Containment  of  CB  agents  within  a  confined  space  allows 
concentrations  to  rapidly  reach  and  sustain  lethal  levels 

•  CB  agents  are  effectively  transported  throughout  a  building 
by  mechanical  systems 

•  Population  densities  are  high 
in  buildings 

•  Agents  can  be  delivered  covertly 

•  Numerous  adsorbing  surfaces 
that  make  building  restoration 
difficult 


COST 


Range  of  Protection  Solutions 


Millions — 


Thousands — 


Hundreds — 


Sheltering 
in  Place 


High  Efficiency 
Filtration 


Expedient 

Protection 

Devices 


Integrated  Passive 
&  Active  Protection 
Systems 


Evacuation 


n  i 

Low  Medium 

LEVEL  OF  PROTECTION 


High 


DARPA  Immune  Building  Overview 


•  Objective:  To  make  military  buildings  less  attractive 
targets  for  attack  with  CB  weapons 

-  Protect  human  occupants 

-  Restore  the  building  to  function  quickly  after  an  attack 

-  Preserve  forensic  evidence  for  medical  treatment  and  retaliation 

•  Protect  all  parts  of  the  building  against  internal  and 
external  releases  of  a  wide  range  of  agents 

•  IB  Program  Accomplishments 

-  Developed  a  highly  effective  building  protection  system 

-  Extensively  tested  protection  system  and  subsystems  in  a  full-scale 
test  bed 

-  Installed  and  demonstrated  system  design  in  an  operational  building 


System  Process  Flow 


Threat  and 

Vulnerability 

Assessment 


\  Protection 

/  Concepts 

Technology 

Development 


Technology 

Development 

Testing 


Test  Bed  Experimental 
Analysis  of  Alternatives 


Preliminary 

Design 


Threat  and  Vulnerability  Assessment 


•  Threat  and  Vulnerability  Assessments  (TV As)  are 
performed  to  identify  requirements  for  building 
protection  systems 

-Threat  Scenarios  were  client  defined: 

-  Agent  Types  -  Exposure  Limits 

-  Release  Masses  &  Locations  -  Environmental  Conditions 

-  Functional  subsystems  were  developed  to  counter  these 
threats 

-  Filtration/Neutralization  -  Detection  and  Forensics 

-  Segmentation  -  HVAC  Responses 


Protection  Concepts 


•  TVA  Outputs: 

-Testable  requirements 

-Technology  development  needs 

-  Foundation  for  initial  system  protection  concepts 


Initial  protection  concepts  were  developed  based  on 

the  requirements  of  the  TVA. 

-  Extensive  modeling  analysis 
performed  to  down-select  the 
most  promising  strategies 

-Generated  an  initial  Test  Bed  design 

-  Defined  interfaces  for  technology 
development  insertions  into  the  system 
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Technology  Development  Testing 


•  Key  areas  underwent  small  scale 
testing/optimization  prior  to 
integration 

-  Distributed  CB  Sampling  System 
-Wall  Leakage  Specifications 

-  Passive  and  Active  Agent  Removal 


-Chemical  Forensics  Sampler 
-Vestibule  Testing 

•  Generated  construction 
requirements 

•  Technologies  tested  in  a  full  scale 
building  and  further  optimized 


Immune  Buildings  Test  Bed  Facility 


•Test  Bed  constructed  in  former  barracks  building  at  Fort 
McClellan  in  Anniston,  AL 


-  Three  stories  with  a  quarter  basement,  ~  30,000  ft2 

-  Entire  building  used  in  Integrated  Systems  Experimentation  phase;  top  two 
floors  only  in  Demonstration  phase 

-  Multiple  HVAC  zones  with  various  protection 
strategies  possible 

-  Performed  over  250  full  scale  building  experiments 


CONTAM  Model  Schematic 
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Test  Bed  Experimentation 


Testing 

-  4  Simulants  to  represent  CB  threats 

-  Methods  to  create  repeatable  releases  of 
simulants  were  developed 


-  Automated  sampling  network 

-  Whole  building  coverage 

-  3  types  of  collectors 

-  Remote  control  of  simulant  release 
and  sample  collection 

Analysis 

-  On-site  laboratory  for  chemical  analysis 

-  Optical  analysis  of  particulate  simulants 

-  Simulant  to  agent  correlations 

-  Data  analysis  methods  (including  uncertainty 


analysis) 


Program  Metrics 


•  Metrics 


-  Fraction  of  Building  Exposed  (FBE) 

-  Fraction  of  Occupants  Exposed  (FOE) 

-  Life-cycle  Cost 


Modeling  /  Experimentation  Process 


Modeling  and  Simulation 


•  Every  test  performed  during  the  Immune  Building 
Program  was  modeled  prior  to  experimentation 

•  CONTAM  modeling,  predicted  the  flow  of 
contaminant  throughout  the  building 

-  Used  to  determine  the  optimum  sampling  locations 

-  Generated  data  for  alternate  agents  and  mass  releases 

-  Generated  data  for  locations  where  releases  were  not 
possible 

•  Test  data  were  used  to  verify 
and  improve  model 
performance 


Design  Modification  and  Phase  II  Test 
Bed  Testing 


•  The  Phase  I  testing  results  guided  the  modifications 
from  the  protection  concept  to  the  preliminary  design 

•  The  Test  Bed  was  reconstructed  to  represent  the 
Demonstration  building  (preliminary  design) 

•  Over  100  Tests  were  performed,  results  were 
gathered  on  the: 

-Overall  system  protection 

-  Subsystem  performance 

-  Effects  of  human  transport  on  the  protection  system 


Design  Optimization 


•  The  final  design  was  generated  based  on  the  results 
of  the  Preliminary  Design  testing 

-The  Test  Bed  was  modified  during  testing  to  reflect  design 
changes  as  they  occurred 

-The  Final  Design  components  were  tested  in  the  Test  Bed 

•  The  final  design  was  installed  in  the  Demonstration 
building 

-Applications  of  Lessons  Learned  from  the  Test  Bed 
allowed  for  an  expedient  commissioning  and 
characterization  process 

-  Performance  Testing  showed  little  deviation  from  the  Final 
Test  Bed  design 
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Conclusions  /  Results 


•  The  Immune  Building  program  employed  a  T&E 
centric  approach  to  developing  designs  per  good 
Systems  Engineering  practice 

•  Data  gathered  in  early  stages  of  the  design  process 
allowed  optimization  prior  to  installation  avoiding 
costly  post-construction  modifications. 

•  Integrating  T&E  into  all  stages  of  the  design  process 
created  a  system  that  was  verified  through  testing  to 
meet  client  requirements. 

•  End  result  is  a  state  of  the  art  system  that  provides 
the  highest  level  of  protection  against  CBR  threats. 
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Overview 

•  Background 

•  Project  Overview 

•  Work  to  Date 

•  Way  Forward 


10th  Annual  Systems  Engineering  Conference 


Background 


•  M&S  Acquisition/T&E  Mission  -  Enable  the  Department  of  the  Navy 
to  effectively  use  M&S  within  and  across  the  Acquisition  Enterprise 

-  Need  a  unified  approach  for  enabling  the  workforce  to  determine  WHICH  tools 
to  use,  WHEN  to  use  them,  and  HOW  to  use  them  across  development 
lifecycle 

-  Need  education  and  options  to  improve  workforce  capabilities  to  select  and  use 
M&S  tools  effectively  and  efficiently.  These  include 

•  Initial  education  and  training,  refresher  training,  continuing  education,  and 
certification  opportunities  once  in  a  career  path 

•  Ultimate  Goal:  M&S  savvy  DoD  acquisition  workforce 

-  Able  to  apply  M&S  tools  appropriately  to  enhance  warfighting  capability, 
reducing  lifecycle  development  time  and  costs. 


10th  Annual  Systems  Engineering  Conference 
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Spiral  One:  Requirements 


•  Emphasis: 

Developing  and  refining  the  needs  assessment  and  performance  metrics 
•  Identify  partner  requirements  (Joint  Curriculum  Definition) 

Content  requirements 

Individual  KSA  assessment  and  knowledge  mapping  tool 
Instructional  Vehicle  Delivery  specifications 
Guidelines  linking  training  content  to  knowledge  gaps 

•  Methods  will  include: 

State  of  the  art  assessment-  Cross  Service 

Task  Analysis:  Content  requirements/System  capabilities 

•  Deliverable:  Learning  Matrix 

Integrates:  Individual  educational  background,  learning  style,  and  workforce  role,  and  desired  education  end  state 


Curriculum  Needs  Assessment 


Specify  Instructional  Vehicle 


Training 
/  Nevtjs 
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Distance  Learning 


0 
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Tools 


0 


Learning  Matri)T 


•Education  gap 
analysis 

•  Knowledge  Mapping 
•M&S  curriculum 
requirements 
•Instructional 
Delivery  System 
specs 
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Stakeholder  Group 


Consists  of  members  from  throughout  DoD 

-  DASN  RDT&E 

-  AFAMS 

-  HQDA 

-  CVN 

-  SPAWAR 

-  COMOPTEVFOR 

-  Future  Combat  System 

Embodies  broad  educational  discipline  representation 
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Market  Segmentation 


Educating  the 
Acquisition  and  T&E 
Workforce  in  the  More 
Effective  Use  of  M&S: 

Market  Schema 


Training  Levels 

Executive 
Management 
Application 
General  Awareness 


Acquisition  Career  Fields 

Program  Management 
Systems  Engineering 
Test  and  Evaluation 
Contracting 
Logistics 

Facilities  Engineering 
Auditing 

Science  &  Technology 

Information  Technology 

Business,  cost  estimating,  and  financial  mgmt 

Industrial  and/or  contract  property  management 

Manufacturing,  production  and  quality  assurance 

Purchasing 
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Information  Trade  Space 
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Identified  Gaps 


As  a  result  of  the  Gap  Analysis  we  conducted,  four  gaps 
were  found  in  the  area  of  workforce  development: 

•  Lack  of  clearly  articulated  competency  statements. 

•  Lack  of  a  widely  accepted  disciplinary  specification  or  body  of 
knowledge. 

•  Lack  of  structured  implementation  of  training  and  education 
vehicles. 


•  Lack  of  a  widely  applied  process  for  certifying  professionals 
based  on  a  community-accepted  disciplinary  specification. 
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High  Level  ESR  Development 


Process: 

-  Initial  list  of  ESR's  developed  by  stakeholders  and  NPS  inter-disciplinary  team. 

-  Stakeholders  involved  in  iterative  process  to  expand  and  refine  ESR’s. 

Results: 

-  17  Process  ESR’s  -Focused  on  the  process  of  choosing  when  to  use  which 
models  and  simulations. 

-  9  Acquisition  ESR’s  -Focused  on  applying  M&S  in  the  acquisition  lifecycle. 

-  5  Test  and  Evaluation  ESR’s  -Focused  on  the  role  and  use  of  M&S  in  test  and 
evaluation. 

-  5  Operational  ESR’s  -Focused  on  the  use  of  operational  and  logistic  M&S  to 
support  Acquisition/T&E  activities. 

-  14  Engineering  ESR’s  -Focused  on  the  use  of  engineering  models  to  support 
Acquisition/T&E  activities. 


10th  Annual  Systems  Engineering  Conference 


Sample  ESRs  of  all  Disciplines 


PI)  Understand  the  critical  decisions  in  the  acquisition  lifecycle,  the 
analysis  plans  to  support  them,  and  the  information  required. 

A2)  Understand  the  concepts  of  Simulation-Based  Acquisition  (SBA) 
across  the  entire  program  life  cycle,  in  order  to  reduce  the  time, 
resources,  and  risks  associated  with  the  acquisition  process. 

T2)  Integrate  M&S,  live  test,  prototype  data,  historical  data,  component 
data,  and  scale  model  data  into  a  coherent  testing  decision. 

04)  Understand  abstractions  and  lower  levels  of  realism  in  operational 
and  logistics  models. 

E2)  Fluid  Dynamics  and  Weapon  System  -  Understand  the  basics  of 
computational  fluid  dynamics  for  CFD  application  and  use  for  M&S. 
Fluid  dynamics  of  subsonic  and  supersonic  weapons,  warheads  and 
their  effects. 


10th  Annual  Systems  Engineering  Conference 
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Spiral  Two:  Component  Development 


•  Emphasis:  Use  the  Learning  Matrix  to  create  necessary  components  for  delivering  training 

-  Content 

-  Instructional  delivery  technologies 

-  Enable  reuse  and  scalability 

•  Methods  will  include: 

-  Blending  SE  with  ISD 

-  Design,  Develop,  Implement  and  Test  components 

•  Deliverables: 

-  Validated  system  components 

•  Content 

•  Delivery  methods 

-  Learning  Architecture  Framework  to  support  integration 


Learning  Matrix  ) 


•Education  gap 
analysis  ™ 

•Knowledge  Mapping 
•M&S  curriculum 
requirements 
•Instructional 
Delivery  System  DOIZ 

specs 

_ J 


Set  of  validated 
components 
and  delivery 
techniques  / 
methodologies 


> 

J 
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Academic  Partners 


•  Air  Force  Institute  of 
Technology 

•  Defense  Acquisition 
University 

•  George  Mason  University 

•  Johns  Hopkins  University/ 
Applied  Physics  Lab 


•  Old  Dominion  University 

•  Stevens  Institute 

•  Texas  A&M 

•  University  of  California, 
San  Diego 

•  University  of  Central 
Florida 


10th  Annual  Systems  Engineering  Conference 
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Course/Module  Concept 


Goal  -  Develop  Course/Module  “Syllabi” 

Syllabi  outline  desired  content  of  educational  elements  that  will  satisfy  the  needs 
identified  in  the  Learning  Matrix. 

Syllabi  combined  into  a  consolidated  and  cohesive  Learning  Architecture. 

Each  module  developed  to  highest  level  of  competency  required  for  the  subject  matter 
(not  always  mastery) 

Modules  constructed  so  that  slices  of  the  content  can  be  extracted  for  lower  required 
competency  levels 

Courses  built  to  target  audience 

Desired  length  of  courses  and  competency  levels  required  determine  subset  of  modules 
combined  into  course  structure 

Human  Capital  Strategy  survey  feedback  will  help  guide  requirements. 


PI  1 

- 1 - 1 

General  Awareness  ( i.e.  W  hrs) 

1 - 1 

PI  .2 

1  1 

PI  3 

1 _ 1 

PI  .4 

1  1  1 

- 1 - 1 - 

Understand  ( i.e.  X  hrs) 

1 - 1 - 1 

PI  .5 

- 1 - 1 - 1  i 

1 _ 1 _ 1 

PI  .6 

1  1 

Application  ( i.e.  Y  Hrs) 

=L _ 1 _ 1 

PI  .7 

PI  .8 

1  1 

Mastery  ( i.e.  Z  hrs) 

PI  .9 

i_ i 

1  1  - 
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Workforce  Mapping 


Mapping  of  ESRs  to  workforce  needs  (Learning  Matrix) 

Performed  by  Academic  Partners,  including  GMU,  JHU/APL,  ODU,  UAH, 
UCF,  and  UCSD 

Three  pieces  provided  to  complete  mapping: 

Workforce  segmentation  definitions 

•  Career  Fields  -  Project  Managers,  Systems  Engineers,  and  T&E  workforce 

•  Career  Levels  -  Basic/entry,  intermediate/journeyman,  and  advanced/senior 
career  levels 

•  Follows  DoD  5000. 52M  descriptions 

Competence  Levels 

•  Four  competence  levels  defined  and  mapped  to  Bloom’s  taxonomy  -  General 
Awareness,  Understand,  Application,  and  Mastery 

Detailed  ESR’s  -  High  level  ESR’s  decomposed  into  “mappable”  level  of 
granularity 
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Program  Management 


Positions  Held: 

-  All  of  functions  of  a  PMO  or  PEO 

-  Program  integrators  and  analysts,  program  managers,  PEOs,  and 
deputies 

-  Support  and  management  positions  throughout  the  workforce 

Responsibilities: 

-  Balance  the  factors  that  influence  cost,  schedule,  and 
performance 

-  Interpret  and  tailor  application  of  the  DoD  5000  Series  regulations 

-  Ensure  that  high-quality,  affordable,  supportable,  and  effective 
defense  systems  are  delivered  to  the  warfighter  as  quickly  as 
possible 
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PM  Career  Levels 


Basic/Entry 

-  Member  working  in  PM  support  role 

-  Example  jobs  include  R&D  coordinator,  test  officer  staff  officer, 
integrator,  analyst,  etc. 

Intermediate/Journeyman 

-  Managers  of  PEO/PMO  office  functions 

-  Deputy  PM  or  PM  for  small  programs,  PEO  staff  roles 

Advanced/Senior 

-  ACAT 1  or  2  PM,  PEO 
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Competence  Levels 


Competence 

Level 

Bloom’s 

Taxonomy 

Definition 

Examples  and  Keywords 

General 

Awareness 

Knowledge 

Recall  or  recognize  data  or 
information. 

Examples:  Recite  a  policy.  Quote  prices  from  memory  to  a 
customer.  Knows  the  safety  rules. 

Keywords:  defines,  describes,  identifies,  knows,  labels,  lists, 
matches,  names,  outlines,  recalls,  recognizes,  reproduces, 
selects,  states. 

Understand 

Comprehension 

Understand  the  meaning, 
translation,  interpolation, 
and  interpretation  of 
instructions  and  problems. 
State  a  problem  in  one's 
own  words. 

Examples:  Rewrites  the  principles  of  test  writing.  Explain  in  one's 
own  words  the  steps  for  performing  a  complex  task.  Translates  an 
equation  into  a  computer  spreadsheet. 

Keywords:  comprehends,  converts,  defends,  distinguishes, 
estimates,  explains,  extends,  generalizes,  gives  Examples,  infers, 
interprets,  paraphrases,  predicts,  rewrites,  summarizes, 
translates. 

Application 

Application 

Use  a  concept  in  a  new 
situation  or  unprompted 
use  of  an  abstraction. 

Applies  what  was  learned 
in  the  classroom  into  novel 
situations  in  the  work  place. 
Put  theory  into  practice, 
use  knowledge  in  response 
to  real  circumstances 

Examples:  Use  a  manual  to  calculate  an  employee's  vacation 
time.  Apply  laws  of  statistics  to  evaluate  the  reliability  of  a  written 
test. 

Keywords:  applies,  changes,  computes,  constructs, 
demonstrates,  discovers,  manipulates,  modifies,  operates, 
predicts,  prepares,  produces,  relates,  shows,  solves,  uses. 

References: 

http://www.nwlink.com/-donclark/hrd/bloom.html 

http://facuitv.washington.edu/krumme/guides/bioomi  .htmi  10th  Annuel  Systems  Engineering  Confefence 
http://www.businessballs.com/bloomstaxonomyoflearningdomains.htm 
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Competence  Levels 


Competence 

Level 

Bloom’s 

Taxonomy 

Definition 

Examples  and  Keywords 

Mastery 

Analysis 

Separates  material  or 
concepts  into  component  parts 
so  that  its  organizational 
structure  may  be  understood. 
Distinguishes  between  facts 
and  inferences. 

Examples:  Troubleshoot  a  piece  of  equipment  by  using  logical 
deduction.  Recognize  logical  fallacies  in  reasoning.  Gathers 
information  from  a  department  and  selects  the  required  tasks  for 
training. 

Keywords:  analyzes,  breaks  down,  compares,  contrasts, 
diagrams,  deconstructs,  differentiates,  discriminates,  distinguishes, 
identifies,  illustrates,  infers,  outlines,  relates,  selects,  separates. 

Synthesis 

Builds/develops  new 
structures,  systems,  models, 
approaches,  or  patterns  from 
diverse  elements.  Put  parts 
together  to  form  a  whole,  with 
emphasis  on  creating  a  new 
meaning  or  structure. 

Examples:  Write  a  company  operations  or  process  manual. 

Design  a  machine  to  perform  a  specific  task.  Integrates  training 
from  several  sources  to  solve  a  problem.  Revises  and  process  to 
improve  the  outcome. 

Keywords:  categorizes,  combines,  compiles,  composes,  creates, 
devises,  designs,  explains,  generates,  modifies,  organizes,  plans, 
rearranges,  reconstructs,  relates,  reorganizes,  revises,  rewrites, 
summarizes,  tells,  writes. 

Evaluation 

Make  judgments  about  the 
value  of  ideas  or  materials. 
Assess  effectiveness  of  whole 
concepts  in  relation  to  values, 
outputs,  efficacy,  viability; 
critical  thinking,  strategic 
comparison  and  review. 

Examples:  Select  the  most  effective  solution.  Hire  the  most 
qualified  candidate.  Explain  and  justify  a  new  budget. 

Keywords:  appraises,  compares,  concludes,  contrasts,  criticizes, 
critiques,  defends,  describes,  discriminates,  evaluates,  explains, 
interprets,  justifies,  relates,  summarizes,  supports. 

http://www.nwlink.com/-donclark/hrd/bloom.html 
http://facultv.washinqton.edu/krumme/quides/bloom1  .html 
http://www.businessballs.com/bloomstaxonomyoflearninqdomains.htm 
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Path  to  Focused  Learning 


Career  Field 


Program  Manager 


Systems  Engineering 
Test  &  Evaluation 


Career  Level 

Basic/Entry 

Intermediate/ 

Journeyman 


Advanced/Senior 


V 


Acquisition 


Test  & 
Evaluation 


Competence  Level 

General  Awareness 


Understand 


v 


Operations/ 

Logistics 


Application 
Mastery 

t 

Engineering 


w 


ESRs 

PI  P2  P3  P4  P5  P6  P 7  P8  P9  P10  P11  PI 2 


P14  P15  P16  P17 


P13.2 

P13.3 

PI  3.4 

P13.5 

P13.6 

P13.7 

P13.8 

P13.9 

Workforce  Mapping  Example 

Learning  Matrix  for  one  ESR  (of  50) 


P13:  Understand  the  trades  between  using  a  general  model  and  a  custom  model,  including  the  VV&A  implications. 

P13.1 

P13.2 

P13.3 

P13.4 

P13.5 

P13.6 

P13.7 

P13.8 

P13.9 

PM 

Basic 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

Intermediate 

Understand 

Application 

Application 

Application 

Application 

Application 

Application 

Mastery 

Mastery 

Advanced 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

SE 

Basic 

Understand 

Understand 

Understand 

Under: 

1_ 1  \irkr\r\irrs+r\inr\ 1 

1_ 1  IryrlrMm+ryyyri_ 1 

1  1  1 

1  1  1 

1_ 1  1  m  

PI  3.1  Define  general  model  and  custom  model 

PI  3.2  State  advantages  of  general  model 

PI  3.3  State  disadvantages  of  general  model 

PI  3.4  State  advantages  of  custom  model 

PI  3.5  State  disadvantages  of  custom  model 

PI  3.6  State  VVA  requirements  of  general  model 

PI  3.7  State  VVA  requirements  of  custom  model 

PI  3.8  Describe  situations  where  each  type  of  model  is 
more  appropriate 

PI  3.9  Given  historical  examples  of  each,  describe  and 
analyze  which  is  more  appropriate 

Intermediate 

Understand 

Application 

Application 

Applici 

Advanced 

Understand 

Application 

Application 

Applici 

T&E 

Basic 

Understand 

Understand 

Understand 

Under: 

Intermediate 

Understand 

Application 

Application 

Applici 

Advanced 

Understand 

Application 

Application 

Applici 

Way  Forward 


•  Spiral  Three  -  Course  Development 

-  Capitalize  on  Academic  Partner  Experience  &  Assets 

-  Continue  to  integrate  Stakeholder  feedback 

-  Ensure  flexibility  in  course  design  through  modular 
concept  (plug  and  play) 

•  Spiral  Four  -  Education  Program  Deployment 

-  Test  Courses  with  student/sponsor  feedback 

-  Implementation  of  Continuous  Assessment  Tool 
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NAVAL 

POSTGRADUATE 

SCHOOL 


Questions? 


Backup  Slides 
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Curriculum  Design 


Many  choices  exist 

-  Ad  Hoc  Approach 

-  Linear  Process 


-  Feedback  Loop  Driven 


Design 


HSH 


Implement 


-  Systems  Engineering  Approach 

-  Instructional  System  Design 

•  ADDIE  phases 


#  Analyze 


Implement 


Evaluate 


Design 


Develop  * 
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Systems  Engineering 


Familiar  SE  Models 

-  Vee 

-  Waterfall 

-  Spiral 

Five  common  items  to  all 

-  Top-down  view  of  entire  system 

-  Life-cycle  approach 

-  Ensure  requirements  are  right 

-  Iterate  using  feedback  loop 

-  Use  interdisciplinary  approach 


Cultural 
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Political 


Techrological 


\  Social 
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Organizational 


Moral/ 

Ethical 


Emotional 


Assessment  &  Feedback 


US  Military  Academy  Approach 
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Project  Overview 


4  Spirals  (Phases)  make  up  the  Project 

1.  Learning  Matrix 

•  Desired  instructional  content  based  on  ESRs  for  Acquisition  workforce 

•  Integrates  educational  background,  learning  style,  workforce  role,  and  desired 
education  end  state 

•  M&S  Workforce  Education  Gap  Analysis 

2.  Learning  Architecture/lnstructional  Framework 

•  Degree/certificate  programs  and  continuous  learning  modules 

•  Content  modules  (course  syllabi) 

3.  Prototype  Curriculum 

•  Develop  curriculum  from  content  architecture 

•  Deliver  with  endorsement/accreditation  to  DAU,  NPS  and  services 

4.  Assessment 

•  Longitudinal  Curriculum  Effectiveness  Evaluation 
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FY07 


FY08 


FY09 


Decomposition  of  Model  Types 


28 


Tying  it  all  Together 


Stakeholder 

Group 


Army 

NavV  Workforce 

Air  Force  Survey 

Marines 

Acquisition 

Training  _  , 

Review  of 

Planning  Consolidated 

BOK 

Testing 

Analysis 

Experimentation 


Learning  Matrix/ 
Instructional 
Content 


M&S  Human 
Capitol  Strategy 


M&S  Body  of 
Knowledge 
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Learning 

Architecture/ 

Instructional 

Framework 


Prototype 
Educational 
Elements  for 
Acquisition 


Application  of 
Educational 
Elements  to  Other 
Communities  and 
Services 


Spiral  Three:  System  Integration  and  Delivery 


•  Emphasis:  Spiral  three  will  create  prototype  curriculum 

-  Modeled  after  other  DAU  courses  like  the  Acquisition  courses  which  have  on  line  and 
schoolhouse  components  based  on  user’s  career  needs 

•  Methods:  The  curriculum  will 

-  Provide  tailorable  learning  modules 

-  Support  various  accreditation  approaches 

-  Leverage  distance  learning  and  schoolhouse  instructional  paradigms. 

•  Deliverable:  Instruction  provided  through  existing  DoD  channels  identified  in 
conjunction  with  DAU 


Spiral  3  will  produce 
validated,  reusable  course 
content  that  can  be 
accessed  by  individuals  at 
various  stages  of  career 
development 


Education 

Gap* * 

Analysis 


Curriculum 

Reqs 


Hr awtedfl* 
•...Maying 


Content 
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Spiral  Four:  Longitudinal  Curriculum 
Effectiveness  Evaluation 


Emphasis:  Spiral  four  will  provide  assessment  and  validation  of  the  long  term  impact 
of  the  curriculum 

Methods:  Base  evaluation  on  Kirkpatrick’s  ‘four  levels’ 

Deliverable: 


-  Measurement  of  the  degree  to  which  this  approach  enhances  performance 

-  Suggestions  for  enhancements  and  modifications 

Using  ISD,  evaluation  cuts  across  all 
Spirals,  but  in  different  ways 


Anafyn  Tmin*ig 

DM-Ign 

Utveigp 

Implement 

Evaluav 

Nt«lj 

T 

^  _ V _ 

, 

Kirkpatrick’s  4 

•  Reactions 
Learning 
Behavior 
Results 


Integrated  Feasibility 
Assessment  1 

•  SME  Review 


Integrated 
Feasibility 
Assessment  2-3 

•  SME  Review 

•  Iterative  Usability 


Results  (impact  on  bottom  line): 

•  Increased  productivity 

•  Improved  quality 

•  Reduced  costs 


Behavior: 

•  Are  KSAs  being  used  in  the 
work  environment? 


Learning: 

•  Were  desired  KSAs  advanced  and 
internalized? 

•  Use  Pre/Post  paradigm 


Reactions: 

•  How  did  students  like  the  program? 

•  Did  it  address  perceived  needs? 

•  How  would  they  change  it? 
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ABSTRACT 

The  Navy  M&S  Office  in  conjunction  with  the  Defense  Acquisition  Modeling  and  Simulation  Working  Group 
presented  the  Naval  Postgraduate  School  with  an  enormous  challenge  in  2006:  design  and  deliver  an  educational 
program  by  2008,  for  20,000  or  more  acquisition  professionals,  focusing  on  the  effective  use  of  modeling  and 
simulation  in  acquisition.  The  acquisition  workforce  is  central  to  force  transformation,  and  education  is  the  key  to 
transforming  that  workforce.  This  paper  describes  the  processes,  lessons  learned  to  date,  and  assessment  plan  for 
this  project. 

We  applied  a  systems  engineering  approach  to  the  problem  of  curricular  design.  The  resulting  solution  consists  of 
four  spirals.  The  first  spiral  focused  on  defining  the  problem.  We  developed  our  analysis  based  on  factors  such  as 
our  market  segmentation  of  the  acquisition  workforce,  the  current  resources  available,  the  state  of  the  modeling  and 
simulation  body  of  knowledge,  the  desired  educational  outcomes  for  each  market  segment,  and  the  gaps  that  existed 
between  those  outcomes  and  the  existing  resources.  At  each  step  in  the  process,  we  involved  key  stakeholders  from 
the  acquisition,  test  and  evaluation  and  training  communities.  We  describe  the  results  of  this  process. 

In  the  second  spiral,  our  goal  was  to  construct  a  learning  architecture  to  cover  the  gaps  identified  in  the  first  spiral. 
We  describe  the  course  content,  scope,  and  delivery  methods  that  we  determined  based  on  those  needs  from  the  first 
spiral. 

The  results  of  the  first  and  second  spirals,  and  subsequent  lessons  learned,  will  be  the  focus  of  our  discussion  herein. 
We  will  also  briefly  summarize  the  third  and  fourth  spirals,  which  are  currently  underway,  that  involve  course 
design  and  testing  in  the  case  of  spiral  three,  and  delivery  and  assessment  of  the  curriculum  for  spiral  four. 
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on  the  analysis  of  systems,  with  emphasis  on  reliability,  quality,  and  warranty  issues. 
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Mr.  Jarema  M.  Didoszak  is  a  Research  Assistant  Professor  in  the  Mechanical  and  Astronautical  Engineering 
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INTRODUCTION 

The  Department  of  Defense  Modeling  and  Simulation  (M&S)  and  Acquisition  communities 
recognize  the  need  for  education  and  training  in  M&S  across  the  acquisition  workforce  (DoD 
M&S  CO,  2006).  The  desired  educational  program  is  different  than  existing  educational 
opportunities  in  that  it  targets  users  of  M&S  rather  than  developers. 

To  meet  this  need,  the  Naval  Postgraduate  School  applied  a  systems  engineering  approach  to 
develop  a  set  of  curricula.  We  submit  that  this  is  different  from  traditional  curricular  design  in 
that  it  enables  the  production  of  a  better  suited  end  product  by  incorporating  systems  engineering 
principles  that  are  not  inherent  in  typical  curriculum  development  projects.  In  particular,  the 
focus  on  requirements  elicitation  from  external  stakeholders  and  requirements  analysis  presents  a 
unique  emphasis. 

The  design  process  incorporated  several  institutions  that  agreed  to  deliver  a  common  set  of 
curricula  to  meet  the  needs  of  the  Defense  community.  This,  too,  is  an  uncommon  practice  in 
curricular  development. 

This  paper  reports  our  progress  as  we  near  the  end  of  the  first  year  of  this  multiyear  program.  It 
contains  a  description  of  the  process  and  of  the  deliverables  produced  during  the  first  phase. 
Further  papers  will  describe  the  results  of  the  curricular  development  implementation  of  later 
phases  of  that  process. 

Herein  we  will  first  provide  a  brief  overview  of  curriculum  design  and  systems  engineering 
approaches.  Then  we  will  show  how  we  applied  these  systems  engineering  approaches  to  the 
design  of  a  set  of  modeling  and  simulation  curricula.  We  present  the  requirements  that  our 
process  defined.  We  sketch  our  future  work  to  complete  this  project.  Finally,  we  will  provide 
some  lessons  learned. 


CURRICULUM  DESIGN 

Traditional  curriculum  design  approaches  vary  from  ad  hoc  construction  of  materials  to  systemic 
Instructional  System  Design  (ISD)  approaches  that  generally  follow  the  ADDIE  model:  analyze, 
design,  develop,  implement,  and  evaluate  (Molenda,  2003).  This  has  been  characterized  as  an 
inherently  linear  process  (Bell  and  Lefoe,  1998).  Other  advocates  characterize  it  as  a  feedback 
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loop.  For  example,  Don  Clark  (2006)  presents  the  flowchart  in  Figure  1  to  characterize  ISD. 
Clark  also  presents  a  detailed  breakdown  of  the  tasks  to  be  performed  in  each  of  the  five  phases 
of  ADDIE. 

Iterative  design  process  appears  in  the  literature.  Bell  and  Lefoe  propose  a  feedback  loop  in  their 
outcomes  based  on  integrative  and  flexible  delivery  models.  Walkington  (2002)  proposes  a 
similar  feedback  cycle.  However,  most  curricular  development  is  sufficiently  challenging  that 
institutions  settle  for  a  single  pass  through  the  design  cycle.  For  example,  the  Electrical  and 
Computer  Engineering  Department  at  Carnegie  Mellon  spent  four  years  on  a  single  redesign  of 
its  curriculum  (Director  et  al.,  1995). 


+  Analyze 
- 1 - W 


± _ I _ 

Develop  *■ 


Figure  1:  ADDIE  model  of  Instructional  Design,  as  a  flowchart.  Adapted  from  Clark  (2006). 

Engineering  accreditation  is  beginning  to  demand  evidence  of  involvement  from  constituents  in 
the  design  and  development  of  engineering  curricula.  The  Engineering  Accreditation 
Commission  of  the  Accreditation  Board  for  Engineering  and  Technology  (ABET)  seeks  “high 
degree  of  involvement  in  defining  objectives  and  desired  outcomes,  assessment,  and 
improvement  cycles;  (and)  sustained  evidence  of  strategic  partnership  with  all  key  constituents.” 
(ABET,  2002)  The  ABET  is  also  pressing  for  evidence  of  feedback  loops  in  curriculum 
revision. 

At  our  own  institution,  curricular  design  incorporates  a  strong  involvement  from  constituents 
(called  “sponsors”  at  NPS)  in  the  services  and  also  a  biennial  curricular  review  process  involving 
constituents  (NPS,  2003).  We  have  also  participated  in  a  multi-school  educational  franchise, 
called  “Product  Development  for  the  21st  Century.”  This  curriculum  was  jointly  developed  and 
delivered  by  NPS,  MIT,  RPI,  and  the  University  of  Detroit  -  Mercy.  All  of  the  partners  deliver 
the  same  courses,  using  a  common  set  of  syllabi. 

Emerging  best  practices,  then,  favor  strong  constituent  input,  feedback  loops,  and  detailed  work 
breakdown.  These  desired  characteristics  lend  themselves  well  to  the  choice  of  a  Systems 
Engineering  (SE)  methodology  as  a  practical  approach  to  curriculum  design. 
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SYSTEMS  ENGINEERING  APPROACH 

As  a  systems  engineering  department,  we  approached  this  project  inside  the  framework  of  our 
traditional  SE  design  models.  Several  models  representing  the  systems  engineering  process  exist 
(Blanchard  and  Fabrycky,  1998).  They  hold  several  principles  in  common.  First,  take  a  top  - 
down  approach  that  views  the  system  as  a  whole.  Second,  take  a  life-cycle  approach  that 
addresses  all  the  phases  of  the  system  life  in  the  design  process.  Third,  get  the  requirements 
right  at  the  start  of  the  project.  This  involves  careful  coordination  with  stakeholders.  Fourth, 
iterate  using  feedback  loops.  Fast,  use  an  interdisciplinary  approach.  The  waterfall,  spiral,  and 
Vee  methods  all  have  these  five  points  in  common. 

We  use  an  approach  similar  to  the  one  developed  at  the  US  Military  Academy,  presented  here  as 
Figure  2.  We  adapted  these  principles  in  our  approach  to  the  design  of  these  curricula. 
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Figure  2:  Systems  engineering  design  process.  From  Parnell,  Driscoll,  and  Henderson  (2006). 

In  keeping  with  the  SE  design  process  concept,  we  took  a  top-down  look  at  the  problem.  We 
segmented  the  target  student  population  by  career  field  and  level  of  expertise.  Next  we  scoped 
the  project  to  address  three  of  the  thirteen  acquisition  career  fields.  They  were  program 
managers,  system  engineers,  and  test  &  evaluators.  The  three  levels  of  expertise  established  for 
each  of  these  career  fields  were  basic/entry,  intermediate/)  oumeyman,  and  advanced/senior. 
These  were  also  in  alignment  with  the  career  levels  defined  by  DoD  Instruction  5000. 52M  (DoD, 
1995).  We  determined  the  educational  requirements  for  each  of  these  nine  (three  by  three) 
audiences.  We  examined  what  was  available  nationally  to  meet  these  educational  requirements, 
and  defined  the  gaps.  We  then  determined  the  requirements  to  design  various  content  modules 
to  address  those  gaps.  This  differs  from  a  bottom-up  approach,  which  would  have  assembled 
existing  courses  into  a  program. 
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We  took  a  life-cycle  approach  to  the  design  of  the  curricula,  focusing  on  assessment  and 
feedback  mechanisms.  We  found  that  designing  the  business  plan  to  support  the  delivery  of  the 
curricula  presented  the  biggest  challenge.  This  is  discussed  in  the  lessons  learned  portion  of  this 
paper.  We  took  great  pains  to  get  the  requirements  correctly  defined.  This,  too,  is  discussed  in 
detail  in  a  subsequent  section.  We  have  planned  for  iteration  in  the  design  of  the  program,  using 
a  test-fix-test  paradigm. 

Last,  we  assembled  an  eclectic  team  of  instructors  from  many  disciplines  and  institutions, 
practitioners,  educational  designers,  representatives  of  the  user  community,  along  with 
representatives  of  industry.  The  requirements  defined  have  been  reviewed  and  accepted  by  this 
broad-based  team. 

Next  we  developed  a  project  plan  for  the  development  of  the  curricula.  It  is  organized  into  four 
spirals,  presented  graphically  in  Figure  3.  The  first  spiral  consists  of  requirements  definition,  the 
second  is  development  of  the  architecture,  the  third  is  detail  design  and  development,  and  the 
fourth  is  delivery  and  assessment.  As  of  the  writing  of  this  paper,  spiral  one  is  complete  and  we 
are  nearly  complete  with  spiral  two. 


Figure  3:  Project  plan. 

How  does  our  approach  differ  from  traditional  instructional  systems  design?  In  many  ways,  it  is 
similar.  Top-down  approach,  analysis  of  requirements,  and  feedback  loops  are  elements  of  ISD. 
What  distinguishes  our  approach  is  a  matter  of  emphasis  on  requirements,  the  interdisciplinary 
nature  of  the  subject  matter  and  hence  of  the  partners,  and  the  life-cycle  planning. 
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In  our  SE  model,  we  are  now  in  the  Solution  Design  Phase,  having  recently  completed  the 
Problem  Definition  Phase,  with  its  emphasis  on  stakeholder  analysis,  functional  analysis,  and 
value  modeling.  Since  our  curricular  design  answers  to  and  must  be  approved  by  a  set  of  external 
customers,  this  problem  differs  from  the  traditional  curricular  design  where  the  decision  makers 
reside  in  the  institution  delivering  the  instruction.  This,  coupled  with  the  necessity  for  building 
consensus  for  design  and  delivery  across  a  wide  set  of  academic  institutions,  has  driven  us  to  use 
a  systems  engineering  paradigm  over  an  ADDIE  paradigm. 


SPIRAL  ONE 

The  first  spiral  began  in  November  2006.  We  assembled  a  team  at  NPS  from  the  engineering 
school,  the  school  of  operational  sciences,  and  the  business  school  to  explore  how  we  would 
build  an  interdisciplinary  and  inter-departmental  team  to  address  the  request  of  the  sponsor. 
Most  schools  find  team  building  across  such  a  range  of  disciplines  and  institutional  boundaries 
challenging,  and  our  school  is  no  different.  We  agreed  on  a  structure  for  organization,  and 
agreed  to  partition  the  work  for  parallel  development,  with  a  small  steering  committee 
responsible  for  organization  and  integration. 

We  set  a  small  team  to  work  collecting  data  on  existing  educational  programs  that  might  address, 
fully  or  partially,  the  education  of  acquisition  personnel  to  employ  modeling  and  simulation  in 
their  projects.  This  resulted  in  a  catalog  of  existing  programs  in  a  relational  database. 

Concurrently,  we  set  up  another  team  to  develop  the  detailed  educational  requirements  for  each 
of  the  nine  market  segments.  Following  terminology  used  at  NPS,  we  called  them  “Educational 
Skill  Requirements”  or  ESRs  for  short.  We  identified  key  representatives  from  the  user 
communities  in  government  and  industry.  We  also  identified  a  set  of  potential  academic  partners 
for  delivery  and  involved  them  in  the  requirement  setting.  The  ESRs  were  broken  into  five  areas 
which  are  presented  below. 

We  compared  the  results  of  the  ESRs  with  the  catalog,  and  identified  the  key  gaps  extant.  Those 
gaps  are  presented  and  discussed  below.  And  finally,  we  organized  the  audiences,  ESRs, 
existing  programs,  and  gaps  into  a  “learning  matrix.”  This  single  document  summarized  the 
requirements  for  the  curricula  to  address  each  segment. 

Concurrent  In  parallel  with  our  work,  partners  at  the  Air  Force  Agency  for  Modeling  and 
Simulation  were  developing  a  human  capital  strategy  for  the  modeling  and  simulation 
community,  and  defining  a  body  of  knowledge  for  that  group  of  professionals.  Their  work 
complemented  ours  by  addressing  a  different  portion  of  the  defense  workforce. 


CATALOG 

The  catalog  contains  data  from  22  institutions  that  offer  relevant  instruction.  It  contains 
information  about  253  courses.  The  data  is  organized  into  tables,  including  institution, 
programs,  courses,  topics,  learning  objectives.  Cost  data  and  delivery  options  are  included.  The 
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catalog  allows  the  team  to  search  for  courses  that  meet  the  proposed  ESRs,  and  to  then  refine  the 
search  by  parameters  such  as  cost,  delivery  method,  duration,  and  location. 

The  focus  of  existing  education  is  on  those  who  develop  and  analyze  models  and  simulations. 
There  are  few  courses  that  focus  on  those  acquisitions  professionals  who  are  supported  by  M&S 
efforts.  Catalano  and  Didoszak  (2007)  found  “Existing  post-graduate  modeling  and  simulation 
degree  programs  produce  engineers  capable  of  developing  M&S,  rather  than  focus  on  the 
required  knowledge  to  use  the  M&S  for  acquisition.”  They  found  29  courses  that  had  an 
acquisition  focus  and  that  targeted  acquisition  professionals.  These  were  in  most  part  “short 
introductory  courses  of  a  few  days  in  length  providing  a  basic  understanding  of  the  use  of  M&S” 
(Catalano  and  Didoszak,  2007).  There  were  only  nine  of  188  traditional  courses  that  addressed 
any  of  the  objectives  of  the  new  program.  They  also  found  that  the  average  cost  per  course  was 
$1271. 


EDUCATIONAL  SKILL  REQUIREMENTS 

After  consulting  with  our  stakeholders,  we  broke  the  ESRs  into  five  groups:  process,  program 
management,  operations  and  logistics,  test  and  evaluation,  and  engineering.  The  first  group 
addressed  common  M&S  issues  for  the  acquisition  community,  and  the  last  four  addressed  issues 
that  focused  on  the  corresponding  domains  of  application. 

The  process  ESRs  are  presented  in  Table  1.  These  ESRs  have  been  vetted  by  users,  sponsors, 
industry,  academic  partners  and  other  stakeholders.  There  is  wide  agreement  that  they  are 
comprehensive  in  scope.  The  rest  of  our  ESR  groups  focus  on  the  domains  of  application. 
Those  ESRs  are  listed  in  Tables  2-5. 

Table  1:  Process  ESRs 


PI)  Understand  the  critical  decisions  in  the  acquisition  lifecycle,  the  analysis  plans  to  support 
them,  and  the  information  required. 

P2)  Understand  the  role  of  modeling  and  simulation  prior  to  the  concept  decision  to  identify  and 
quantify  capability  gaps,  and  to  estimate  how  well  new  program  concepts  might  address  those  gaps. 
P3)  Understand  the  costs,  benefits,  and  risks  of  using  physical  testing,  modeling  and  simulation, 
and  historical  data  to  provide  information  for  acquisition  decisions. 

P4)  Know  the  technical  aspects  of  the  domain  of  application. 

P5)  Know  the  taxonomy  and  hierarchies  of  models  and  simulations  and  be  able  to  select 
appropriately  for  a  given  situation.  Understand  the  types  of  architectures  and  role  of  architectures 
in  tying  together  and  communicating  requirements,  analysis,  modeling  and  simulation,  design,  and 
development  planning  to  all  stakeholders.  Understand  how  M&S  is  deployed  in  different 
environments  (Live,  Virtual,  and  Constructive).  Understand  the  differences  between  standalone  and 
confederated  M&S  applications  and  when  to  apply  each  in  various  situations.  Be  familiar  with  the 
simulation  interoperability  standards. 

P6)  Establish  and  write  valid  modeling  and  simulation  requirements  using  a  process  that  includes 
modeling  and  simulation  needs  analysis,  generation  of  valid  modeling  and  simulation  requirements, 
functional  decomposition  and  conceptual  model  development,  and  issuance  of  “built  to”  or  “buy  to” 
performance  specifications.  Understand  how  models  and  simulations  evolve  in  fidelity,  resolution, 
and  scope  as  the  program  life  cycle  progresses. 

P7)  Estimate  the  cost,  develop  a  schedule,  and  measure  the  performance  of  a  modeling  and 
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simulation  plan.  Identify  the  areas  of  risk  and  develop  a  mitigation  strategy. 

P8)  Know  how  to  incorporate  modeling  and  simulation,  through  a  Simulation  Support  Plan,  into  a 
systems  engineering  plan  and  a  test  and  evaluation  master  plan. 

P9)  Know  and  require  the  best  practices  and  standards  in  modeling  and  simulation  as  developed  in 
key  case  studies. 

P 1 0)  Know  the  models  and  simulations  used  in  a  given  domain,  their  inputs  and  outputs,  and  their 
strengths  and  weaknesses. 

P 1 1)  Know  the  common  terminology  and  high  level  roles  and  responsibilities,  as  well  as  the 
underlying  philosophy,  principles,  and  methodologies  used  in  VV&A  efforts,  especially  those 
applied  in  DoD. 

PI 2)  Be  able  to  correctly  match  the  level  of  detail  of  a  model  with  that  of  the  information  needed  to 
support  a  decision,  and  understand  the  connection  between  the  decision  to  be  made  and  the 
estimation  of  measures  from  the  model. 

PI 3)  Understand  the  trades  between  using  a  general  model  and  a  custom  model,  including  the 
VV&A  implications. 

PI 4)  Design  a  sound  simulation  study  for  a  given  set  of  objectives. 

PI 5)  Apply  appropriate  statistical  techniques  to  the  analysis  of  simulation  output. 

PI 6)  Know  how  to  manage  and  reuse  existing  models,  data,  and  simulations  appropriately  and 
assure  that  new  products  developed  are  designed  and  prepared  for  reuse. 

PI 7)  Manage  the  data  strategy  for  an  M&S  effort  including  estimating  the  resources  necessary  to 
obtain  sufficient  data  to  populate  the  model. 


Table  2:  Program  Manager  ESRs 


Al)  Understand  the  types,  role  and  value  of  formal  Modeling  and  Simulations,  and  their  various 
characterizations  for  application  to  systems  management,  particularly  with  respect  to  design,  testing, 
training,  production,  cost  estimation,  manning,  and  logistical  simulations. 

A2)  Understand  the  concepts  of  Simulation-Based  Acquisition  (SBA)  across  the  entire  program 
life  cycle,  in  order  to  reduce  the  time,  resources,  and  risks  associated  with  the  acquisition  process. 
A3)  Be  able  to  discern  among  M&S  proposals,  relative  to  measurable  program  contributions,  and 
decide  on  the  appropriate  program  office  level  of  expenditure  on  M&S  tools  throughout  the 
program  life  cycle.  Distinguish  whether  custom  or  off-the-shelf  products  will  be  best  suited  for  the 
program’s  purpose. 

A4)  Understand  the  role  of  M&S  in  the  contract  proposal  process,  how  M&S  efforts  will  be 
defined  and  specified,  and  the  value  of  M&S  deliverables  under  an  acquisition  contract.  Determine 
their  need  for  continuous  improvement,  vis-a-vis  M&S  cost/benefit  trades  throughout  the  program 
life  cycle. 

A5)  Know  where  to  find  organizational  M&S  resources  to  identify  the  number  and  types  of  models 
currently  in  use,  best  practices  from  case  studies,  where  they  originated,  and  how  they  might  be 
leveraged  in  support  of  an  acquisition  program. 

A6)  Be  aware  of  the  Modeling  and  Simulation  Resource  Repository  as  a  single  source  for 
information  about  and  access  to  DoD  models,  simulations,  data  sources,  algorithms,  and  other  M&S 
resources  in  order  to  facilitate  reuse  and  avoid  duplication. 

A7)  Understand  experimental  design,  level  of  model  detail,  and  M&S  application  as  a  pre-test 
prediction  tool.  Use  M&S  to  make  informed  engineering  tradeoff  analyses  through  the  program’s 
Decision  Risk  Analysis  process.  Understand  the  analysis  of  M&S  outputs/measures. 

A8)  Understand  the  critical  interrelationships  and  balance  between  modeling  and  simulation  and 
more  traditional  forms  of  test  and  evaluation  (T&E)  -  particularly  operational  and  live-fire  test  and 
evaluation. 

A9)  Know  how  to  employ  M&S  to  explore  reliability  and  interoperability  issues. 


Table  3:  Test  and  evaluation  ESRs 
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Tl)  Quantify  the  risk  of  using  M&S  in  place  of  live  testing.  For  open  systems,  quantify  the  risk  of 
using  M&S  to  evaluate  a  single  system  component  in  place  of  testing  an  entire  configuration. 

T2)  Integrate  M&S,  live  test,  prototype  data,  historical  data,  component  data,  and  scale  model  data 
into  a  coherent  testing  decision. 

T3)  Understand  the  different  types  of  testing  (i.e.  unit,  integration,  interoperability,  and 
operational)  and  identify  the  utility,  limitations  and  risks  for  use  of  M&S  in  each. 

T4)  Understand  the  potential  opportunities  for  employing  M&S  in  the  test  planning  and  execution 
process. 

T5)  Be  aware  of  existing  M&S  T&E  facilities  used  within  the  DoD. 


Table  4:  Operational  and  logistical  modeling  ESRs 


01)  Understand  the  role  of  operational  and  logistical  models  in  the  acquisition  life  cycle  and  when 
they  are  used. 

02)  Know  the  properties  of  a  representative  suite  of  operational  models  across  the  services, 
including  required  inputs,  outputs,  assumptions,  implementation  requirements,  costs,  time  required, 
adaptability  and  extensibility,  and  VVA  status. 

03)  Know  the  properties  of  a  representative  suite  of  logistical  models  across  the  services, 
including  required  inputs,  outputs,  assumptions,  implementation  requirements,  costs,  time  required, 
adaptability  and  extensibility,  and  VVA  status. 

04)  Understand  abstractions  and  lower  levels  of  realism  in  operational  and  logistics  models. 

05)  Understand  and  be  able  to  model  the  components  of  logistics  systems, 
including  Supply  Chain,  Storage  systems,  Facilities,  Production,  Inventory 
management,  Transportation  &  distribution,  Replenishment  policies. 


Table  5:  Engineering  ESRs 


Depending  on  the  system  being  acquired,  a  particular  subset  of  these  may  apply: 

El)  Structural  Mechanics,  Shock  and  Vibrations  -  Understand  basic  structural  mechanics 
including  stress-strain  relations,  buckling  and  fatigue,  shock  and  vibration,  and  finite  element 
methods  in  M&S. 

E2)  Fluid  Dynamics  and  Weapon  System  -  Understand  the  basics  of  computational  fluid 
dynamics  for  CFD  application  and  use  for  M&S.  Fluid  dynamics  of  subsonic  and  supersonic 
weapons,  warheads  and  their  effects. 

E3)  Dynamics  and  Control  -  Understand  the  basics  of  M&S  in  process  and  multi-physics 
(mechanical,  electrical  &  hydraulic)  based  dynamic  system  controls. 

E4)  Thermodynamics  and  Heat  Transfer  -  Understand  the  fundamentals  of  thermodynamics  and 
heat  transfer  with  applications  to  M&S  in  engineering  power  cycles,  propulsion  and  auxiliary 
system  cycle  analysis  and  design. 

E5)  Materials  and  Fabrication  -  Possess  a  basic  understanding  of  the  materials  technology 
associated  with  manufacturing,  welding  and  corrosion  control.  Have  an  introduction  to  composite, 
superconducting  materials,  and  fiber  optics  as  applied  to  M&S. 

E6)  Acoustic  and  Electromagnetic  Systems  -  Have  a  general  awareness  of  the  fundamentals  of 
acoustic  and  electromagnetic  wave  propagation  and  application  to  DoD  systems. 

E7)  Military  Platform  Systems  Engineering  -  Appreciate  the  broad-based  design  oriented  M&S 
approach  for  complex  platforms  that  interact  with  air-land-sea-based  hardware  systems,  command 
and  control  systems  and  combat  systems. 

E8)  Computers  -  Recognize  basic  computer  system  architecture,  operating  systems,  networking 
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and  introduction  to  engineering  software  and  their  applications.  Possess  at  least  a  limited 
proficiency  in  a  structured  programming  language  such  as  Fortran  or  C,  and  be  able  to  use  such 
tools  for  code  development.  Gain  exposure  to  finite  element/difference  codes,  with  application  to 
solve  engineering  problems  including  experience  with  selected  software  packages. 

E9)  Electrical  Engineering  -  Understand  basic  circuit  analysis  including  DC  and  AC  circuits.  Gain 
an  exposure  to  the  construction  and  operating  characteristics  of  rotating  machinery,  static 
converters,  power  distribution  systems  and  multi-phased  circuits. 

E10)  C4ISR  -  Understand  the  requirement  for  Command,  Control,  Communications  Computers, 
Intelligence,  Surveillance  and  Reconnaissance  in  systems.  Understand  the  basic  components, 
methods  and  alternatives  for  transferring  information  from  one  point  to  another  both  internal  and 
external  to  the  system  being  considered.  Have  the  ability  to  analyze  all  available  technologies  for 
achieving  rapid/effective/jam-resistant  information  transfer. 

Ell)  Networks  -  Understand  the  principles  of  networks  applied  to  military  applications  including 
physical,  command  and  control,  and  social  networks  and  their  implications  for  engineering  design 
of  system 

El 2)  Environment  -  Understand  the  fundamentals  of  terrestrial  science  (geology,  oceanography, 
meteorology,  and  near-earth  space  science)  to  describe  how  systems  interact  with  and  are  influenced 
by  their  environment. 

El 3)  Human  Systems  Integration  -  Understand  the  principles  of  Human  Systems  Integration. 
Describe  the  applications  of  M&S  to  support  HSI  design  and  analysis. 

El 4)  Aerodynamics  -  Understand  the  principles  of  aerodynamics  with  applications  to  M&S. 
Understand  the  cost,  schedule,  and  iterative  development  nature  of  simulation  testbeds  used  for 
flight  software  development  through  formal  qualification. 


These  process  ESRs  contain  several  noteworthy  tasks.  They  indicate  that  the  integration  of 
modeling  and  simulation  as  a  source  of  data  into  formal  decision  making  processes  remains  an 
important  challenge  for  acquisition  professionals.  P5  requires  the  appropriate  selection  of  a 
model  and  simulation  for  a  given  situation.  P6  requires  the  student  to  establish  and  write  valid 
modeling  and  simulation  requirements.  P7  requires  the  student  to  demonstrate  project 
management  skills  for  M&S  activities,  including  cost  estimation,  scheduling,  performance 
assessment,  and  risk  identification  and  mitigation.  There  was  wide  consensus  that  the  skills  and 
knowledge  identified  in  the  process  ESRs  were  vital,  and  that  it  was  of  great  importance  to 
deliver  these  widely  throughout  the  M&S  workforce. 

The  engineering  ESRs  in  Table  5  also  deserve  special  comment.  We  observed  that  many  in  the 
acquisition  community  had  a  greater  familiarity  with  operational  models  than  with  engineering 
models.  Operational  models  are  useful  for  verifying  that  the  correct  set  of  capabilities  is  defined 
in  the  concept  development  phase.  Engineering  models  are  useful  for  design,  and  especially  for 
testing.  In  fact,  if  one  desires  to  substitute  M&S  results  for  live  testing,  one  is  most  often 
contemplating  the  use  of  an  engineering  model. 

After  long  discussion  and  careful  consideration  of  the  audience,  we  decided  that  formal  survey 
courses  on  the  principles  listed  in  Table  5  was  not  going  to  be  palatable  to  the  general  members 
of  the  acquisition  community,  who  lacked  the  time  and  background  to  complete  them 
successfully. 

We  decided  to  address  the  engineering  ESRs  through  a  set  of  case  studies  that  provide  the 
engineering  context  as  they  presented  the  case.  Accordingly,  we  commissioned  preliminary 
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design  of  eleven  case  studies.  These  range  from  the  dynamics  and  control  theory  underlying  the 
Segway  machine,  to  the  structural  mechanics,  fluid  mechanics,  and  environmental  science 
behind  ship  shock  simulation  models. 


RESULTS  OF  GAP  ANALYSIS 

The  gap  analysis  revealed  two  main  gaps.  First,  where  there  were  courses  and  material  that 
addressed  the  ESRs,  there  was  no  common  look  and  feel  to  them.  In  their  current  state,  they 
cannot  be  easily  integrated  into  a  coherent  whole.  For  example,  the  Defense  Acquisition 
University  has  a  module  on  M&S  for  System  Engineering  (DAU,  2006)  that  is  delivered  on-line. 
A  second  short  course  is  offered  by  George  Mason  University  in  a  three  day,  21  hour  short- 
course  delivered  in  traditional  lecture  format.  These  courses  cover  some  of  the  ESRs  but  not  at 
the  depth  necessary  for  some  of  the  audiences.  It  is  not  possible  to  integrate  the  two  courses  as 
they  exist  as  they  were  not  designed  for  such  integration  and  since  they  use  different  modes  of 
delivery. 

The  second  main  gap  was  that  a  number  of  key  ESRs  had  no  courses  or  materials  that  addressed 
them  at  the  level  desired  for  the  acquisition  professionals.  This  was  particularly  true  of  the 
engineering  ESRs.  It  was  also  true  for  several  of  the  program  manager  and  process  ESRs. 

A  detailed  report  on  the  gap  analysis  is  available  upon  request  to  the  authors. 


PARTNERSHIP  PROCESS 

The  target  audience  for  these  curricula  is  estimated  at  20,000  students.  This  exceeds  the  capacity 
of  any  one  educational  institution.  To  address  this,  we  recruited  partner  schools  from  across  the 
nation  to  participate  in  the  project.  Partners  as  of  this  writing  include:  George  Mason  University, 
Johns  Hopkins  University  /  Applied  Physics  Lab,  Old  Dominion  University,  Stevens  Institute  of 
Technology,  University  of  Alabama  (Huntsville),  University  of  California  (San  Diego),  and  the 
University  of  Central  Florida.  We  have  divided  work  among  ourselves  according  to  our  specific 
competencies  and  strengths.  For  example,  the  University  of  Alabama  (Huntsville)  has  a  national 
reputation  for  its  simulation  based  testing  work,  and  that  school  volunteered  to  lead  the  design 
work  for  many  of  the  T&E  ESRs  previously  shown  in  Table  3. 

We  developed  a  metaphor  for  our  approach.  We  consider  ourselves  a  national  food  franchise 
chain.  Together  with  the  academic  partners,  we  are  designing  a  menu  and  a  store  layout.  All  the 
institutions  can  open  their  own  franchise  store,  but  the  layout  and  menu  will  be  standardized 
across  the  chain.  Quality  benchmarks  and  standardized  syllabi  will  help  assure  that  the  product 
at  one  store  is  the  functional  equivalent  of  the  product  at  any  another  store. 

We  have  broad  agreement  on  this  approach,  but  as  we  have  not  yet  completed  design  integration, 
the  level  of  difficulty  in  bringing  this  approach  into  reality  remains  an  open  question. 
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STAKEHOLDER  FEEDBACK 

Our  panel  of  stakeholders  includes  representation  across  the  services.  The  Navy  is  represented 
by  staff  from  the  Secretary  of  the  Navy’s  office,  the  Naval  Air  Systems  Command,  the  Space  and 
Naval  Warfare  Systems  Command,  and  the  Commander  Operational  Test  and  Evaluation  Force. 
The  Army  is  represented  by  staff  from  Headquarters,  Department  of  the  Army,  and  the  Future 
Combat  System  Program  Office.  The  Air  Force  is  represented  by  the  Air  Force  Agency  for 
Modeling  and  Simulation  and  the  Joint  Strike  Fighter  Program  Office.  The  Marine  Corps  has 
been  represented  by  staff  from  the  Expeditionary  Fighting  Vehicle  Program  Office.  Industry 
has  been  represented  by  Boeing. 

We  have  iterated  the  approach  and  the  ESRs  several  times  through  the  stakeholders  to  achieve 
consensus.  We  have  also  briefed  the  senior  members  of  the  Defense  Modeling  and  Simulation 
Coordination  Office  on  progress  to  date,  and  we  have  incorporated  their  feedback  as  part  of  our 
design  process. 

Major  design  reviews  are  scheduled  at  the  end  of  each  spiral  with  representatives  of  the 
stakeholders  and  the  sponsors. 


SPIRAL  TWO 

Our  current  spiral  takes  the  results  of  our  gap  analysis  and  develops  syllabi  for  content  modules 
to  address  those  gaps.  The  majority  of  this  work  is  being  performed  by  our  academic  partners 
and  is  nearly  complete.  Much  of  this  work  focuses  on  defining  the  detailed  ESRs  which  will 
then  in  turn  create  the  learning  architecture  through  an  index  of  specific  tasks  fulfilling  the  stated 
educational  requirements.  An  example  of  what  the  detailed  ESRs  might  look  like  for  P13,  one  of 
the  Process  ESRs  from  Table  1,  is  shown  in  Figure  4. 

Here  the  overarching  ESR  is  decomposed  into  sublevel  ESRs.  The  depth  of  knowledge  for  each 
of  the  career  fields,  at  the  accompanying  career  level,  shown  as  you  enter  the  table  from  the  left, 
is  defined  by  the  general  competence  levels:  General  Awareness,  Understand,  Application  and 
Mastery.  Once  complete,  each  of  the  50  ESRs  will  have  a  corresponding  table  consisting  of 
detailed  ESRs  mapped  to  the  appropriate  level  of  granularity.  This  then  forms  the  consistent  and 
cohesive  structure  of  our  learning  architecture. 
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P13:  Understand  the  trades  between  using  a  general  model  and  a  custom  model,  including  the  VV&A  implications. 

P13.1 

P13.2 

P13.3 

P13.4 

P13.5 

P13.6 

P13.7 

P13.8 

P13.9 

PM 

Basic 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

General 

Awareness 

Intermediate 

Understand 

Application 

Application 

Application 

Application 

Application 

Application 

Mastery 

Mastery 

Advanced 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

Understand 

SE 

Basic 

Understand 

Understand 

Understand 

Und 

PI  3.1  Define  general  model  and  custom  model 

PI  3.2  State  advantages  of  general  model 

PI  3.3  State  disadvantages  of  general  model 

PI  3.4  State  advantages  of  custom  model 

PI  3.5  State  disadvantages  of  custom  model 

PI  3.6  State  VVA  requirements  of  general  model 

PI  3.7  State  VVA  requirements  of  custom  model 

PI 3.8  Describe  situations  where  each  type  of  model  is 
more  appropriate 

PI  3.9  Given  historical  examples  of  each,  describe  and 
analyze  which  is  more  appropriate 

Intermediate 

Understand 

Application 

Application 

Appl 

Advanced 

Understand 

Application 

Application 

App 

T&E 

Basic 

Understand 

Understand 

Understand 

Und 

Intermediate 

Understand 

Application 

Application 

Appl 

Advanced 

Understand 

Application 

Application 

Appl 

Figure  4:  Example  of  Sublevel  ESRs  and  Corresponding  Career  Level  Competencies. 

While  each  module  will  be  developed  to  the  highest  level  of  competency  that  is  required  for  the 
subject  matter,  it  may  not  necessarily  be  to  the  Mastery  level.  By  implementing  this  kind  of  a 
methodology  in  the  design  of  the  course  modules,  portions  of  content  containing  more  details 
information  can  later  be  extracted  to  meet  lower  required  competency  levels  without  a  major 
overhaul  of  the  course. 


This  concept  allows  for  the  flexibility  in  creating  courses  that  are  tailored  to  the  target  audience, 
one  of  the  key  stakeholder  inputs  stressed  heavily  during  our  early  discussions  on  the 
deployment  of  this  education  program.  Any  number  of  desired  competency  levels,  course 
lengths  and  delivery  methods  can  then  be  combined  to  provide  an  optimized  solution  in 
educating  the  end  user. 

As  expected,  Spiral  Two  will  conclude  with  a  design  review  where  our  sponsors  will  approve  the 
work  prior  to  moving  to  the  next  stage. 


WAY  AHEAD:  SUBSEQUENT  SPIRALS 

Our  instructional  design  team  at  NPS  will  complete  the  templates  for  the  “common  look  and 
feel”  for  the  content  modules.  The  engineering  case  studies  that  were  previously  mentioned  will 
be  one  of  the  first  deliverables  and  will  serve  as  a  first  opportunity  to  “test  market”  our  product. 
As  theses  case  studies  will  be  used  to  support  the  courses  created  in  Spiral  Three,  we  anticipate 
incorporating  this  initial  feedback  in  early  to  make  the  greatest  impact  on  the  course 
development  process. 

At  present  we  are  planning  a  mixture  of  traditional  academic  courses,  short  courses,  online 
courses,  stand-alone  reference  material,  and  collections  of  case  studies.  We  will  confirm  with  our 
stakeholders  the  delivery  methods  that  will  be  used. 


Olwell,  Johnson  &  Didoszak  /  Application  of  Systems  Engineering  Principles  in  the  Design  of  Acquisition  Workforce  Curricula  Page  13  of  16 


NDIA  l(fh  Annual  Systems  Engineering  Conference  2007 


As  part  of  the  life-cycle  analysis,  we  will  develop  a  long-term  business  model  with  our  sponsors. 
The  model  will  account  for  delivery,  maintenance,  and  periodic  update  costs. 

Spiral  Three  includes  the  actual  development  of  the  content  modules  and  supporting  materials. 
It  includes  a  classroom  test  of  the  materials,  and  the  generation  of  feedback  on  those  materials 
from  students,  sponsors,  and  stakeholders.  Following  that  feedback,  one  quarter  is  allocated  for 
corrections  to  address  any  deficiencies  and  to  disseminate  any  exceptional  best  practices. 

Spiral  Four  is  the  production  delivery  and  the  longitudinal  assessment  of  the  effectiveness  of  the 
material. 


A  NOTE  ON  ASSESSMENT 

Three  separate  assessment  efforts  are  underway.  The  first  is  to  assess  student  knowledge  before 
and  after  completion  of  the  content  for  his  or  her  market  segment.  This  involves  a  pre-test  that 
will  also  be  used  to  tailor  material  to  the  student.  At  the  end  of  the  curriculum,  a  post-test  will 
assess  the  student’s  mastery  of  the  educational  skill  requirements.  The  pre-test  and  post-test  are 
being  developed  at  NPS  for  web-delivery. 

The  second  is  an  assessment  of  the  appropriateness  of  the  educational  skill  requirements.  This 
will  involve  long-term  surveys  of  both  graduates  and  their  employers  for  feedback  on  how  useful 
the  ESRs  were  in  the  performance  of  their  duties. 

The  third  assessment  effort  will  focus  on  the  effectiveness  of  the  instruction  and  will  be 
administered  to  students  at  the  completion  of  each  content  module.  This  will  be  used  to 
continuously  improve  the  delivery  of  the  information. 


LESSONS  LEARNED 

Lessons  learned  to  date  are  necessarily  preliminary.  Nonetheless,  we  have  some  initial  findings. 
It  has  taken  longer  to  build  consensus  among  the  wide  group  of  stakeholders  represented  than  we 
originally  anticipated.  Thus,  the  greater  the  number  of  partners,  the  less  agile  the  effort 
becomes.  There  is  an  enormous  amount  of  coordination  and  synchronization  necessary  in  an 
undertaking  such  as  this.  This  requires  much  greater  management  than  we  had  anticipated. 
Obtaining  consensus  is  also  difficult  when  team  members  have  different  visions. 

The  business  plan  cannot  be  ignored  when  building  curricula.  When  we  started  this  project,  the 
initial  business  plan  was  that  the  costs  of  delivery  would  be  centrally  funded.  This  changed  to  a 
customer-funded  model  as  we  got  underway.  The  mechanics  of  that  funding  and  revenue 
sharing  are  being  worked  out.  Of  greater  importance,  unless  the  workforce  is  presented  with 
incentives  to  enroll  in  the  curricula,  there  is  a  risk  of  low  enrollment.  The  sponsor  bears  the 
responsibility  to  help  create  demand  in  the  acquisition  workforce,  as  that  is  beyond  the  scope  of 
the  project.  The  sponsor  is  considering  adding  the  completion  of  the  content  of  this  program  to 
the  credentials  necessary  to  advance  in  the  acquisition  workforce.  This  also  involves  risk,  since 
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there  are  many  stakeholders  involved  in  the  management  of  the  qualifications  of  the  acquisition 
workforce,  and  the  M&S  CO  is  but  one  of  them.  The  risk  to  the  academic  partners  has  been 
mitigated  by  paying  them  the  full  cost  to  develop  their  materials,  but  there  is  still  risk  to  DoD;  if 
we  “build  it,  but  they  do  not  come.” 

Integration  is  emerging  as  a  challenge.  The  curricula  must  be  vertically  and  horizontally 
integrated.  We  acknowledge  that  there  is  risk  when  there  are  so  many  different  delivering 
institutions  involved.  Our  mitigation  strategy  is  to  provide  detailed  templates  and  regular 
feedback.  This  still  promises  to  be  a  challenge. 


CONCLUSION 

This  project  is  immensely  challenging.  We  believe  that  the  only  way  it  can  be  successfully 
completed  is  to  apply  basic  systems  engineering  principles  to  the  design  and  execution  of  the 
curricular  design.  We  are  taking  a  top-down  approach,  addressing  the  curricular  system  as  a 
whole.  We  are  also  taking  a  life-cycle  approach.  We  have  diligently  worked  to  establish  the 
educational  skill  requirements,  and  we  are  developing  the  delivery  requirements  as  of  this 
writing.  We  have  structured  feedback  loops  into  the  program.  Last,  we  have  built  a  team  that  is 
inter-disciplinary,  inter-departmental,  and  inter-scholastic. 

The  requirements  that  we  have  identified  are  an  important  step  towards  the  improvement  of  the 
acquisition  workforce,  the  better  implementation  of  M&S  in  acquisition  activities,  and  the 
continuing  transformation  of  the  way  the  acquisition  enterprise  does  business. 

This  effort  has  been  noted  as  a  model  for  other  Defense  educational  initiatives.  In  particular,  the 
re-engineering  of  the  Navy  Systems  Engineering  training  and  education  strategy  is  being  based 
on  a  template  derived  from  this  approach. 
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Preface  +  3  Distinct  Sections-94  Pages 


SEP  Prep  Guide 

History 


•  Version  1 .02  released  in  February  2006 
(31  pages) 

•  Describe  application  of  SE  in  the  various  life  cycle 
phases 

•  Provide  information  to  specific  questions  for  each 
of  the  CR/TD,  SDD/Production,  and  Sustainment 
phases 


SEP  Prep  Guide 

Reasons  for  Update 

•  SEP  quality  was  inconsistent 

•  ‘Lessons  Learned’  from  PSRs 

•  Feedback  from  SEP  Reviews 


SEP  Prep  Guide 

Update  Process 

•  ED  released  a  draft  SEP  Prep  Guide  on  27  April  for 
review  by  the  SE  Forum  members  and  other  SE 
personnel. 

-  600  comments  received  and  adjudicated 

•  ED  released  a  second  draft  on  25  July 

-  close  to  1 00  additional  comments  received  and  adjudicated 

•  Released  Version  2.0  of  the  SEP  Prep  Guide  on  18 
October  07  (Preface  +  3  distinct  sections-94  pages) 


SEP  Prep  Guide 

Goals 

•  Provide  clear  and  unambiguous  guidance 
on  SEP  preparation  with  lessons  learned 

•  Assist  the  SEP  Preparation  Team  by 
tailoring  sections  for  Acquisition 
Milestones  A,  B  and  C 

•  Prompt  the  SEP  Preparation  Team  to 
consider  key  planning  factors  in  each 
focus  area 


It’s  About  Technical  Planning. ..Not  the  Document 


SEP  Prep  Guide 

Update  Details 

•  New  guide  includes  sections  by  program  phase: 

-  Milestone  A/T echnology  Development 

-  Milestone  B/System  Development  &  Demonstration 

-  Milestone  C/  Production  &  Deployment  and 
Operations  &  Support 

•  Each  section  is  based  on  technical  planning  focus  areas 
for  that  phase 

-  Program  Requirements 

-  Technical  Staffing 

-  Technical  Baseline  Management 

-  Technical  Review  Planning 

-  Integration  with  Overall  Management  of  the  Program 


SEP  Prep  Guide 

Update  Details 


•  Program  Requirements 

-  Describe: 


•  Desired  capabilities  and  traceability  to 
requirements 

•  Statutory  and  regulatory 

•  Specified  and  derived 

•  Certification 

•  Design  considerations 


SEP  Prep  Guide 

Update  Details 

•  Technical  Staffing 

-  Describe: 

•  Lead  Systems  Engineer/Functional  roles 

•  IPT  Organization/Structure 

•  IPT  Staffing 

•  IPT  Coordination 

•  Integration  with  the  Contractor  and  External 
Organizations 


SEP  Prep  Guide 

Update  Details 

•  Technical  Baseline  Management 

-  Describe: 

•  Who  is  responsible  for  technical  baseline 
management 

•  Approach  to  defining,  approving  and  maintaining 
the  baseline 

•  Allocation  and  verification  of  program  requirements 

•  Alignment  between  the  specification  tree  and  the 
WBS 

•  Assessment  of  technical  maturity 


SEP  Prep  Guide 

Update  Details 

•  Technical  Review  Planning 

-  Describe: 

•  Event-driven  technical  reviews 

•  Technical  review  management 

•  Chairing  of  technical  reviews 

•  Stakeholder  participation  in  technical  reviews 

•  Peer  participation  in  technical  reviews 


SEP  Prep  Guide 

Update  Details 

•  Integration  with  Overall  Management  of 
the  Program 

-  Describe: 

•  Linkage  to  other  program  management  plans 
(Acquisition  Strategy,  IMP,  IMS,  EVM,  Risk,  etc) 

•  PM’s  approach  to  technical  reviews 

•  Risk  management  approach 

•  Integration  of  T&E 

•  Integration  with  Sustainment 

•  Integration  of  SE  considerations  into  the  contract 


The  Way  Ahead 

Communicate/Implement 


•  Communicate  the  new  SEP  Prep  guidance 
to  Government  and  Industry 

•  Implement: 

-  Initial  SEPs  submitted  for  MDA  approval  shall 
be  IAW  SEP  Prep  Guide  Version  2.0  on  1  Jan 
08 

-  Updated  SEPs  submitted  for  MDA  approval 
shall  be  IAW  SEP  Prep  Guide  Version  2.0  on 
1  June  08 


The  Way  Ahead 

Update  the  DA  G 


Incorporate  new  SE  Policy  in  DoD  5000.2 


•Enclosure  12.  Includes  new  policy  on  CM,  DM,  and  ESOH  and  previously  approved  SE  and  related 
policies. 

•  Enclosure  3.  Table  E3.T2.  SEP  is  mandated  at  milestones  A,  B,  and  C. 

•  §  3.5.5.  SE  “shall  be  considered”  during  CR  and  TD. 

•  §  3.7.7.  “System  Design  [phase  of  SDD]  shall  include  the  establishment  of  the  functional,  allocated,  and 
product  baselines  for  all  configuration  items.” 

•  §  3.7.8.  Proceeding  beyond  the  CDR.  “The  system-level  CDR  provides  an  opportunity  for  mid-phase 
assessment  of  design  maturity  as  evidenced  by  measures  such  as  successful  completion  of  subsystem 
CDR;  the  percentage  of  hardware  and  software  product  build-to  specifications  and  drawings  completed 
and  under  configuration  control.” 

•  §  3.7.9.  System  Demonstration.  “The  program  shall  enter  System  Demonstration  when  the  program  has 
successfully  completed  the  system-level  CDR  and  established  an  initial  product  baseline.” 

•  §  3.10.5.  Program  Support  Reviews  (PSRs)  mandated  for  all  MDAPs  and  “.  .  .  shall  be  conducted  prior  to 
each  milestone  event,  before  approval  of  the  SDD  acquisition  strategy,  and  at  other  times  as  directed  by 
the  USD(AT&L).” 


The  Way  Ahead 

SE  Plan  Unification 


Unified  SE  Plan 


Working  Towards  a  Unified  Systems 

Engineering  Plan 


NDIA  10th  Annual 
Systems  Engineering  Conference 

October  24,  2007 


Col.  Richard  Hoeferkamp 
DoD  OUSD  A&T  (SSE) 


Robert  (Bob)  Scheurer  P.E.,  P.M.P. 
Boeing  Integrated  Defense  Systems 


Purpose 


Present  findings  of  SE  Working  Group  discussion 
between  ODUSD  (A&T/SSE)  and  Boeing  on 
Acquirer  -  Supplier  technical  planning 


Background 


Mission: 


“Define  the  environment  within  which  SEP/SEMP 
unification  can  be  enabled,  agreed  upon,  and  executed. 
These  documents,  which  may  be  initially  separate,  will 
be  unified  into  a  single  document  by  the  time  of  IBR  but 
still  link  to  related,  subordinate  documents  that  are  likely 
specific  to  the  Acquirer  and  Supplier.” 


Progression  of  Discussions 


SEP  -  SEMP  alignment 

•  Similarities  and  differences  between  Acquirer  &  Supplier  SE  Plans 

•  Intent  of  both 

•  Gaps  and  misalignments  between  the  two 

•  Influences  of  existing  policy  &  guidance,  incl.  DID  81024  Systems 
Engineering  Management  Plan 

Migration  from  alignment  of  “SEP  /  SEMP”  to  unification 

•  Pros  and  cons  of  Acquirer  /  Supplier  unification:  from  Pre-RFP  to  Post¬ 
contract  award 

•  Unified  SE  planning  phasing 

Methodology  towards  a  unified  approach 

Acquirer:  DoD 

Supplier:  Prime  Contractor  and  its  Suppliers 


DoD  SEP  Prep  Guide 
&  Sample  Supplier  SEMP 


SEP  Prep.  Guide  2nd  Edition  VI  .0 

1.0  Introduction 


Supplier  SEMP 

1.0  Introduction 

>  2.0  Applicable  &  Reference  Documents 
3.0  Program  Overview 


2.0  Program  Reqmts 


4.0  SE  Organization  Integration  &  Tech  Authority 


3.0  Technical  Staffing  &  Org.  Planning. 


4.0  Technical  Baseline  Management 


5.0  Technical  Review  Planning 


6.0  Integration  with  Overall  Management  of  the  Program 


5.0  SE  Technical  Processes 
6.0  SE  Mgt.  Processes 

_  -r 
✓ 

✓ 

* 

7.0  Integ.  With  Prog.  Mgt.  Efforts 

H 

(  8.0  Transitioning  Critical  Technologies 
9.0  Integ.  Environment  for  Sys.  Engineering 
10.0  Additional  SE  Activities 
1 1 .0  Supporting  Plans 
12.0  Notes  y 
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Respective  Guides  Address  Many  Common  Topics 


Why  Not  A  Single  SE  Plan?  I-"™"" 


Pros  : 

•  Common  vision 

•  Acquirer/Suppliers  with  stronger  team  emphasis 

•  Shared  responsibilities 

•  Clear  understanding  of  programmatic  &  technical  planning 

-  Drives  alignment  of  program  support  documents  (IMP/IMS, 
SOW,  WBS,  PEP,  TEP,  etc.) 

•  Potential  downstream  cost  avoidance  &  schedule  savings 

Cons: 

•  Cultural  changes  (i.e.  not  accustomed  to  a  unified  SE  Plan) 

•  Additional  up  front  planning  time 

•  The  challenge  of  achieving  greater  communication  between 
Acquirer  /  Supplier 

•  Potential  contractual  ramifications  when  updating  plan 

•  Lack  of  detailed  implementation  /  experience  on  both  sides 

•  Potential  increase  in  contract  proposal  costs? 


Vision:  SE  Plan  Unification 


•  Acauirer/Supplier-developed  technical  plan  for  SE  implementation 

•  Acquirer/Supplier  shared  roles  and  responsibilities  in  SE  effort 

•  Acquirer/Supplier  conducted  event  driven  technical  reviews 

•  Acquirer/Supplier  teaming  on  linkage  with  other  program  plans 


Path  to  a  Unified  SE  Plan 


Unified 


SE  Plan 


Acquirer 
SE  Pian(s) 


Supplier-Specific 
Lower-  Level 
Planning 


l  i 

Update 

Update  Event  2 

Event  1  SRR 

IBR 


Update 
Event  n 


Aligned 
SE  Plans 


Bidders’ 

Conference 


Proposal 

Submittal 


Contract 

Award 


MSG07-1 1 61 94-009ppt 


Unified  SE  Planning  Phasing 

Notional  Draft  for  Typical  SDD  Program 


Alignment/  Unification 


Contract 


Schedule 


Acquirer  SE  Plan 


Initial  SE  Plan 
(Draft  SEMP) 


Bidders9 

Conference 


RFP 


Unified  SE  Plan 


j 

Proposal 

Submittal 


Contract 

Award 


Update 
Event  1 

IBR 


Update 
Event  2 

SRR 


Update 
Event  n 


Phase  I 

Leading  Up  to  Bidders’  Conference 


Situation: 

Acquirer  requirements  emerging  for  upcoming  acquisition 
Supplier  technical  solutions  or  solution  components  evolving  (potentially 
independently  developed) 


Activities: 

Acquirer  developing  draft  SEP  for  Bidders’  Conference  (for  prospective 
Supplier(s)  feedback) 

Prospective  Supplier(s)  developing  draft  SEMP  for  emerging 
proposed  technical  solutions 


Alignment/  Unification 


Unified  SIE  Plan 

1 


Update  Update 
Event  1  Event  2 

[BR  SRR 


Update 
Event  n 
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Phase  lla 


Post-Bidder’s  Conference  to  Proposal  Submittal 


Situation: 

Suppliers  have  awareness  of  Acquirer’s  Draft  SEP  content 


Activities: 

Acquirer  making  updates  to  SEP  based  on  Bidder’s  Conference 
feedback 

Supplier  making  modifications  to  Draft  SEMP 
and  lower  tier  plans  for  alignment  to  SEP 


Initial  Supplier  identification  of 
integration  issues,  risks,  and 
opportunities  in  the  proposal 


Alignment/  Unification 


Contract 


Schedule 


Acquirer  SE  Plan 

Draft  SEP 


Supplier  SE  Plan 

Initial  SE  Fla  i 
(Draft  SEMP 


Bidders1 

Conference 


lib  |  III 

Update 


Unifiiecl  SE  Plan 

▼ 


RFP 

Proposal 

Submittal 


Contract 

Award 


Update  Update 
Event  1  Event  2 

[BR  SRR 


Update 
Event  n 
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Phase  Mb 

Post-Proposal  Submittal  to  Contract  Award 


Situation: 

Acquirer  will  have  received  SEMP(s)  from  the  Supplier(s) 

A  generally-quiet  period  from  the  standpoint  of  Acquirer-Supplier 
planning  interaction 


Activities: 

Acquirer  evaluating  Suppliers’  SEMP(s),  associated  plan  artifacts 
(IMP/IMS),  and  quality  of  the  Acquirer  planning  in  the  proposal. 
Acquirer  makes  prospective  modifications  to  SE  Plan  framework  per 
selected  Supplier  SEMP 
Supplier(s)  performing  implementation 
preparations  in  anticipation  of 
ATP  /  Contract  Award 
Additional  integration  issues,  risks, 
and  opportunities  identified 


Contract 


Schedule 


Supplier  SE  Plan 

Initial  SE  Plan 
(Draft  SEMP) 


RFP  j 

Proposal 

Submittal 


Update  Update  Update 

Event  1  Event  2  Event  n 

[BR  SRR 
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Phase  III 

Post-Contract  Award  to  IBR 


Situation: 

Many  opportunities  for  planning  changes 

Time  for  the  most  significant  plan  unification  to  occur,  all  under  the 
constraints  of  the  contract 


Activities: 

Substantial  discussions  regarding  the  push  toward  a  unified  SE  Plan. 
Both  Acquirer  and  Suppliers  adjusting  planning  to  better  align  with 
contractual  commitments 
Integration  issues  resolved  and  mitigation/ 
realization  plans  developed 
More  specific  planning  details 
emerging  and  established 


Alignment/  Unification 


Acquirer  SE  Plan 

Draft  SEP 


Contract 


Schedule 


±3. 


lla  i  lib 


Supplier  SE  Plan 

Initial  SE  Plan 
(Draft  SEMP) 


Unified 

— 

|  SE  Plan 

!V 

Bidders1 

Conference 


RFP 


Update  Update 
Event  1  Event  2 

IBR  SRR 


Update 
Event  n 


MSG07-1 161 94-01 5.ppt 


Phase  IV 

Post-IBR 

Situation: 

Unified  SEP 


Activities: 

Unified  plan  being  implemented/executed,  w/success  dependent  upon 
how  well  the  contract  was  structured. 


Results  of  program  execution  may  vary  (w/plan  variation)  depending  on 
whether  the  program  contract  is  “Cost  Plus”  or  “Firm-Fixed  Price” 

Opportunities  being  identified  for  next 

Unified  SE  Plan  Update  Event 


Contract 


Schedule 


Supplier  SE  Plan 

Initial  SE  Plan 
(Draft  SEMP) 


RFP 

Proposal 

Submittal 


Contract 

Award 


Update  Update 
Event  1  Event  2 

[BR  SRR 


Update 
Event  n 


MSG07-1 161 94-01 6.ppt 


Summary  of  Phasing 


•  Unification  requires  reference  to  OSD  “SEP  Prep  Guide,” 
“Integrating  SE  with  DoD  Acquisition  Contracts  Guide,” 
and  relevant  industry  SEMP  Guide 

•  RFPs  will  include  Acquirer  (Govt.)  SE  Plan 

•  Unified  SE  Plan:  A  single  unified  technical  planning 
document  detailed  down  to  a  specified  level  that 
integrates  (after  contract  award)  the  SEP  (Govt  SE  Plan) 
in  the  RFP  with  the  SEMP  (Industry  SE  Plan)  in  their 
original  proposal 


Threats  to  Plan  Unification 


•  Languaging/Definitions 

•  Organization  and  Cultures 

•  Proprietary  Limitations 

•  Contractual  Constraints 

•  Working  Relationships  of  Participants 


Way  Ahead 


Share  findings  with  Government  &  Industry  Forums  --  solicit  feedback 

•  SE  Forum 

•  NDIA  SE  Conference,  Oct  07 

•  Other 

Coordinate  unified  SE  plan  implementation  details  with  contracting 

Propose  DoD  policy  to  implement  unified  SE  Plan 

Review  various  guides  for  revisions  as  appropriate  pending  policy 
decision 

Update  DID  81024  -  Systems  Engineering  Management  Plan 


Questions? 


•  What  Show  Stoppers  to  this  Concept? 

•  What  Would  be  the  Update  Frequency,  Criteria,  and 
Approval  Authority  for  this  Concept? 

•  How  Might  This  Process  be  Prototyped? 

•  What  Front-End  Guidance  Would  You  Give  to  the 
Acquirer  (e.g.,  DoD)  for  Deploying  this  Concept? 


Backup/Reference  Material 


Working  Group  Approach 


Coordination  between  SE  planning  representatives  of  Govt.  &  Boeing 
Establish  vision  and  scope  (topics)  of  discussion/  engagement 
Share  experiences,  best  practices,  &  lessons  learned 
Identify  implementation/  improvement  opportunities 


Jointly  share  findings  &  recommendations 


SE  Planning  Update  Event  Types 


New  Initiative  (Start  SEMP) 

Authorization  to  Proceed  (ATP) 

Program  Milestone  Reviews 
Periodic/Scheduled  Updates 

Coordinated  with  Other  Significant  Document  Updates 

•  PEP 

•  Subordinate  Documents  (e.g.,  TEP,  RMP,  etc.) 

•  Supplier  Plans 

Leading/Lagging  Indicators  Signaling  the  Need 

•  Tests  (Failures) 

•  Significant  Changes  in  Program  Events/Outcomes 

•  Audit  Results 

•  Comments  from 

Customers 
End  Users 
Program  Personnel 


Current  Findings  /  Consideration^^"47 


Product  of  SE  process  is  technical  baselines  and  reviews 

Emphasis  should  be  on  event-driven  reviews;  e.g.,  in  the  SEP  or  SEMP,  how 
do  you  know  when  you  are  ready  for  CDR? 

Describing  your  program  processes  is  not  equal  to  drafting  your  program 
SEMP  or  defining  your  plan 

An  objective  of  DoD  is  to  eliminate  “Canned  SEPs”  and  “SEPs  for  Hire”. 

A  multi-faceted  approach  is  being  used  by  the  DoD  for  implementing  SE  plans 
(SEPs)  on  programs  and  hopefully  changing  culture: 

•  Issuing  the  SEP  Preparation  Guide 

•  Conducting  Awareness  Training 

•  Performing  Hands-On  Follow-Up  Activities 

Program  managers  need  to  be  more  involved  with  systems  engineering 
Recommendations  by  systems  engineering  must  be  taken  more  seriously. 


SE  Planning  Environment 


Multiple  Participant  Bodies 
Multiple  Teams 
Multiple  Locales 
Multiple  Levels  of  Interest 
Multiple  Time  Spans 


Aligning  &  Unifying  SE  Plans 


Step  1 :  Identify  Plan  Components 


Planning  Context 
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Aligning  &  Unifying  SE  Plans 


Step  2:  Link  Plan  Components 


Acquirer 

Supplier 

SEP 

,  , 

SEMP 

Program  Requirements 

i — / 

System  Capabilities 

Technical  Staffing  and  Org.  Planning 

1 — i 

Organization 

Technical  Maturation  and  Baseline  Mgt. 

SE  Processes  (Technical) 

1 — 1 

SE  Processes  (Management) 

Technical  Review  and  Audit  Planning 

Integration  of  Technical  Effort 

Integration  w/  Over-All  Mgt.  of  Program 

i - A 

Additional  SE  Activities 

i - A 

Supporting  Plans 
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Defining  Software  Component 
Specifications:  An  Open  Approach 


Kenneth  Klein 

Computer  Sciences  Corporation 


'C 


EXPERIENCE.  RESULTS. 


NDIA  Systems  Engineering  Conference 
October  22-26,  2007 


Copyright  ©  2004  Computer  Sciences  Corporation.  All  rights  reserved. 
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A  Couple  Definitions 

•  Open 

-  Based  on  widely  excepted  and  supported  standards 
-Defines  key  interfaces  using  these  standards 

-  Not  proprietary 

•  Software  Component 

-A  modular  part  of  a  software  design  that  hides  its 
implementation  behind  a  set  of  external  interfaces. 

-Within  a  system,  components  satisfying  the  same  interfaces 
may  be  substituted  freely. 


•  That’s  what  the  terms  mean  in  the  context  of  these  slides.... 
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EXPERIENCE.  RESULTS. 


The  Problem 

•  Given  an  “as-built”  component-based  Department  of 
Defense  (DoD)  software  system 

-Code  written  in  Java 

-  Interface-based  component  services 

•  Needed  an  approach  to  documenting  each  component  as  a 
as  a  set  of  well-defined  interfaces 

-Required  to  meet  DoD  “openness”  standards 

-Critical  for  making  components  extendible  and  reusable 
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Well-Defined  is  not  Well-Defined 

•  A  lot  of  literature  available  on  defining: 

-Information  exchange  standards,  e.g.,  CORBA,  JMS,  DDS 
-Specific  implementations  of  these  standards 
-Component  frameworks,  e.g.,  SOA,  EJB 
-Quality  of  Service  requirements 

•  Not  so  much  out  there  on  defining  a  service’s  functional 
behavior 
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The  Solution 

•  Define  the  component  and  its  services  using: 

-Lightweight  UML  domain  modeling 
-Design  by  Contract  (DbC)  principles 

•  Tools  used 

-A  UML  modeling  tool  that  can  generate  HTML  output 
-  Doxygen 

•  Open  source  C++/Java  documentation  generation  tool 

-  Similar  to  Javadoc 

>>  Recognizes  Javadoc  comment  delimiters 

•  Reads  source  code,  generates  HTML 

•  www.doxygen.org 

-A  web  browser 
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Overview 

This  document  provides  the  component  specification  forthe  specified  component.  This  includes  the  overall  component  contract,  as  well  as  the  contracts  for  each  of  the 
component's  services.  A  domain  model  is  also  included. 

How  to  Use  This  Document 

The  overall  component  contract  can  be  found  in  the  Component  Contract  page. 

The  individual  service  contracts  can  be  found  by  first  navigating  to  the  Component  Services  page,  then  clicking  down  to  the  classes  that  contain  the  functions  that  provide  1 
services.  Each  function's  abstract  contains  its  contract.  To  find  a  particular  function,  click  Functions  to  display  an  alphabetical  function  list. 

The  domain  model  is  available  by  selecting  the  Domain  Model  page  and  navigating  through  the  HTML  version  of  the  UML  class  diagram. 
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This  document  provides  the  component  specification  forthe  specified  component.  This  includes  the  overall  component  contract,  as  well  as  the  contracts  for  each  of  the 
component's  services.  A  domain  model  is  also  included. 

How  to  Use  This  Document 

The  overall  component  contract  can  be  found  in  the  Component  Contract  page. 

The  individual  service  contracts  can  be  found  by  first  navigating  to  the  Component  Services  page,  then  clicking  down  to  the  classes  that  contain  the  functions  that  provide  1 
services.  Each  function's  abstract  contains  its  contract.  To  find  a  particular  function,  click  Functions  to  display  an  alphabetical  function  list. 

The  domain  model  is  available  by  selecting  the  Domain  Model  page  and  navigating  through  the  HTML  version  of  the  UML  class  diagram. 
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Context  Diagram 

•  Shows  component’s  provided  and  required  interfaces 

-  Provided  interface  declares  services  that  this  component  offers 
to  external  components 

-  Required  interface  declares  services  that  this  component 
requires  from  external  components 

•  Describes  required  interfaces  in  context  of  this  component 

-Each  component  may  describe  the  same  required  interface 
differently  based  on  the  component’s  needs 

-E.g.,  given  an  Illuminator  interface,  one  client  may  require  it  to 
check  Illuminator  equipment  status ;  another  client  may  require 
it  to  Illuminate  a  target 
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Context  Diagram  Example  (cont.) 
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Component  Services 

•  What  is  expected  of  the  client? 

•  What  does  the  service  do? 

•  State  all  this  with  as  few  implementation  details  as  possible 

-Most  implementation  choices  should  not  impact  the 
specification 

-More  likely  specification  will  remain  unclassified 

•  Design  by  Contract  provides  a  solution 
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What  is  Design  by  Contract  (DbC)? 

•  Defines  the  contract  between  the  interface  and  its  clients 

•  Preconditions 

-States  that  must  be  true  when  service  is  invoked 

•  Postconditions 

-  If  service  is  invoked  when  preconditions  are  true, 
postconditions  describe  guaranteed  outcome 

•  e.g.,  state  changes,  messages  sent 

•  Invariants 

-Attribute  constraints  that  must  always  be  true: 

•  After  component  instantiation 

•  Before/after  each  service  invocation 

•  e.g.,  An  Engagement  must  have  exactly  one  Target 

•  Exceptions 

-Describes  what  happens  when  preconditions  or  invariants  are 
violated  or  postconditions  cannot  be  met 

-Behavior  can  be  “undefined” 
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Why  DbC? 

•  Well  documented,  mature  paradigm 

-Term  coined  by  Bertrand  Meyer  in  1997 
-Since  then,  large  volume  of  literature  written  on  the  topic 
•  See  resource  list 

•  Decoupled  from  implementation  details 

-Guidelines  for  “just  enough”  information 
-Implementation  can  change  without  impacting  contract 

•  Encourages  discussions  that  may  otherwise  never  occur 

-  Provides  common  vocabulary  for  complex  concepts 

-  Exceptions  often  discovered  when  writing  contracts 
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Why  DbC?  (cont.) 

•  Facilitates  Liskov  Substitutability  Principle  (LSP) 

-Service  implementations/extensions  must  not  add 
preconditions  or  remove  postconditions 

•  Supports: 

—  Mai  ntai  n  abi  I  ity/Exte  nd  i  b  i  I  ity/Reu  sabi  I  ity 
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DbC  Note:  Preconditions  and  Callbacks 

•  Callback  services  should  not  change  the  state  that  triggered 
the  callback 

-  Remaining  observers  will  receive  incorrect  notifications 

•  Subject  component  has  a  list  of  color  observers 

•  Subject  reports  “I  just  turned  red” 

•  One  of  the  observers  changes  subject  to  blue 

•  The  remaining  observers  will  incorrectly  be  notified  that  subject  is 
red 

•  Mitigation 

-Subject  component  keeps  track  of  whether  a  callback  is  in 
progress 

-Any  offered  service  that  could  change  an  observed  state  has  a 
precondition  that  notifications  are  not  in  progress 

•  Above  observer’s  attempt  to  make  subject  blue  would  be  rejected 
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DbC  Note:  Maintaining  the  Invariants 

•  Exceptional  service  termination  must  restore  component 
invariants 

-Otherwise,  component  is  not  stable,  so  its  services’  behavior  is 
undefined 

-May  be  criteria  for  invoking  recovery  path 

•  Concurrency  should  only  be  allowed  for  services  that  can 
guarantee  that  preemption  can  only  occur  while  the 
component  invariants  are  in  place 

-Mitigated  by  many  concurrency  oriented  architecture  and 
design  patterns 

•  See  Pattern-Oriented  Software  Architecture  Vol.  2:  Patterns  for 
Concurrent  and  Networked  Objects 
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Component  Service  Contracts 

•  Method  signature 

-Captured  as-is  from  source  code 

•  Preconditions/Postconditions/Exceptions 

•  Query  or  Command 

-  Does  the  service  change  the  parameters’  or  the  component’s 
state? 

•  Parameters 

-Constraints 

•  E.g.,  valid  ranges,  precision,  units 

-  Is  ownership  transferred? 

•  If  “no,”  client  must  be  notified  of  any  state  changes 
-Type  Definitions 

•  Imported  as-is  from  source  code 

•  Linked  via  hypertext 
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Component  Service  Contracts  (cont.) 

•  Quality  of  Service 

-Performance,  throughput,  blocking,  availability 

•  Is  concurrency  allowed? 
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Service  Contracts  as  Comments  in  the  Code 

•  From  The  Mythical  Man  Month  (M3),  pg.  169  [Fred  Brooks,  1995] 
(first  edition  published  in  1975) 

-  We  typically  attempt  to  maintain  a  machine-readable  form  of  a  program 
and  an  independent  set  of  human-readable  documentation,  consistent 
of  prose  and  flow  charts. 

-  The  results  in  fact  confirm  our  teachings  about  the  folly  of  separate 
files.  Program  documentation  is  notoriously  poor,  and  its  maintenance 
is  worse. 

-  The  solution,  I  think,  is  to  merge  the  files,  to  incorporate  the 
documentation  in  the  source  program. 


•  This  is  what  Doxygen  does... 
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Component  Services  Example:  Java  Code 

/** 

*  DESCRIPTION: 

*  <p> 

*  This  method  will  distribute  the  request  to  the  WeaponResourceManager . 

*  <p> 

*  @param[in]  request  The  request  being  sent  to  the  WeaponResourceManager. 

*  -#  Valid  ranges 

*  -  Not  null 

*  @par  Query  or  Command: 

*  Command 

*  @pre 

*  -#  None. 

*  ©post 

*  -#  If  the  request  is  an  Alpha  Request,  it  was  added  to  the 

*  Illuminator  Schedule. 


*/ 

public  void  setRequest(  RequestIF  request  ); 
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Component  Services  Example:  Doxygen  Output 


DESCRIPTION: 


This  method  will  distribute  the  request  to  the  WeaponResourceManager 


Parameters: 

[in]  request  The  request  being  sent  to  the  WeaponResourceManager. 


<2 


1.  Valid  ranges 
o  Not  null 
2!  HVmI  iyi  ol  ll|J  Li  dl  Ibl'eiTed  (Y/N)?  Y 


Query  or  Command: 

Command 

Precondition: 

1.  None. 


Extracted  from 
code  comment 
block 


Postcondition: 

1 .  If  the  request  is  an  Alpha  Request,  it  was  added  to  the  Illuminator  Schedule.  This  processing  includes  the  use  of  the  Illuminator  interface 
to  acquire  equipment  status. 

2.  If  the  request  is  a  Beta  Request,  it  was  added  to  the  Launcher  Schedule.  This  processing  includes  the  use  of  the  Launcher  interface  to 
acquire  equipment  status. 


Blocking  ( 

N 
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Component  Contracts 

•  Preconditions/Postconditions 

-Before  and  after  component  startup,  respectively 

•  Invariants 

-Applicable  to  component  as  a  whole 

•  Each  is  a  pre  &  post  condition  for  every  service 

•  Exceptions 

-  If  pre/post  conditions  or  invariants  violated 

•  “Full  load”  memory  requirements 

•  Proven  hardware  platform  and  OS  support 

•  Communication  standards  and  implementations 

-E.g.,  JMS/Websphere,  DDS/NDDS  4.0,  CORBA/ACE  TAO 

•  Other  Protocols/Standards 

-POSIX,  SNMP,  .NET,  etc. 
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Component  Contracts  (cont.) 

•  Programming  Languages 

•  Configuration  file  dependencies 

•  Availability  requirements,  e.g.,  MTBF 
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Component  Contract  Example 


EXPERIENCE.  RESULTS. 


WeaponResourceManager  Component  Contract 


Precondition: 

The  AlphaProperties  configuration  file  must  exist. 

The  initial  state  must  be  provided. 

Postcondition: 

The  WeaponResourceManager  was  started  and  the  initial  state  was  set  using  the  provided  parameter. 
The  WeaponResourceManager  applied  the  states  found  in  the  AlphaProperties  file. 

Communications  were  established  with  the  Illuminator,  Ship  and  Launcher  components. 


Invariant: 

Communication  must  be  maintained  with  the  Illuminator,  Ship  and  Launcher  components. 


Exceptions: 

MissingPropertiesFile  The  AlphaProperties  file  does  not  exist.  Error  is  logged  ant 

invalidProperties  Properties  in  the  AlphaProperties  file  were  missing  or  invalid. 

RequiredCornoonentsUnavailable  Unable  to  communicate  with  the  Illuminator,  Ship  or  LauncnE 


igram  exits. 

The  properties  are  set  to  their  default  values, 
[ehavior  undefined. 


Standard  Used  for  Invoking  Services 

JMS/TibCO  4.0.0 

Language  Dependencies 

Java  5.0 
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Contract  Ambiguity  Problem 

•  From  M3  pp.  63-64 

-Human  language  is  not  naturally  a  precision  instrument  for 
[specification]  definitions. 

-  Formal  definitions  are  precise.  What  they  lack  is 
comprehensibility. 

-I  think  we  will  see  future  specifications  to  consist  of  both  a 
formal  definition  and  a  prose  definition. 


•  This  is  what  the  Domain  Model  does... 


10/5/2007  10:28:48  AM  28 


Contract  Ambiguity  Solution:  The  Domain  Model 

•  Provides  formality  of  UML 

-Each  domain  class  is  clearly  defined  in  the  model 

•  Contracts  reference  domain  classes  in  plain  English 

•  What  is  a  domain  class? 

-Real-situation  notional  class  in  a  domain,  e.g.,  Launcher 
Schedule,  Target,  etc. 

-They  are  not  actual  software  implementation  classes 

•  Why  not  implementation  classes? 

-  M3  says  (pg.  1 75):  “If  one  uses  only  a  highest-level  structure 
graph,  it  might  safely  be  kept  as  a  separate  document,  for  it  is 
not  subject  to  frequent  change  ” 

-Notional  domain  classes  are  stable,  because  they  are 
decoupled  from  implementation  details 
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Domain  Model  Example 

WeaponResourceManager:  Schedules  and  Scheduled  Events 


<<component>> 

WeaponResourceManager 

+active5tate 


\ 

LauncherSchedule 

0..* 


V0"* 


V 

LauncherScheduleEvent 

IlluminatorScheduleEvent 
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How  Domain  Model  Relates  to  Contracts 


EXPERIENCE.  RESULTS. 


Contract 


Postcondition: 

1 .  If  the  request  is  an  Alpha  Request,  it  was  added  to 
to  acquire  equipment  status. 

2.  If  the  request  is  a  Beta  Request,  it  was  added  tg, 
acquire  equipment  status. 


Class  IlluminatorSchedule  {Analysis} 

Documentation 

The  IlluminatorSchedule  contains  all  of  the  current  illuminator  reservations. 


Parent  Package 

DomainObiects 

Abstract 

No 

10/5/2007  10:28:48  AM  32 


Glossary 


il  Weapon  Resource  Scheduler  Component  Specification  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  lools  Help 

On-  •  O-B  gift | P  Search  Favorites  [S]  •dhT« 

Address  ©  V:\software  engineering\Open  Architecture\Component  Development\How  to  define  components  specif ications\Trusted  download\WR5  5-22\html\index.html 

UNCLASSIFIED 

Weapon  Resource  Scheduler  Component  Specification 

Home  ■  Component  Contract"  Component  Services  ■  Classes  ■  Functions  ■  Domain  Mode 

This  document  provides  the  component  specification  forthe  specified  component.  This  includes  the  overall  component  contract,  as  well  as  the  contracts  for  each  of  the 
component's  services.  A  domain  model  is  also  included. 

How  to  Use  This  Document 

The  overall  component  contract  can  be  found  in  the  Component  Contract  page. 

The  individual  service  contracts  can  be  found  by  first  navigating  to  the  Component  Services  page,  then  clicking  down  to  the  classes  that  contain  the  functions  that  provide  1 
services.  Each  function's  abstract  contains  its  contract.  To  find  a  particular  function,  click  Functions  to  display  an  alphabetical  function  list. 

The  domain  model  is  available  by  selecting  the  Domain  Model  page  and  navigating  through  the  HTML  version  of  the  UML  class  diagram. 


10/5/2007  10:28:48  AM  33 


Glossary  Excerpt 


Component  Contracts 

Component  contracts  apply  to  the  component  as  a  whole.  They  should  not  be  confused  with  Component  Service  Contracts  which 
apply  to  particular  sen/ices. 

Precondition: 

System  and/or  environment  states  required  in  order  to  successfully  instantiate  the  component,  e.g.,  config  files  must  exist. 

Postcondition: 

Component  states  that  exist  upon  completion  of  component  instantiation  and  initialization.  For  OA,  this  includes  completion  of 
the  component's  start  method.  Examples  of  component  level  postconditions  include: 

•  Indicating  if  the  component  was  placed  in  active  or  standby  state 

•  Listing  which  config  files  were  read 

•  Stating  that  communications  with  other  specific  components  were  initialized 

•  Listing  which  subscriptions  and  publications  were  defined 


Invariant: 

Component  states  that  must  be  true  after  component  instantiation/initialization  and  at  the  start  and  completion  of  any  offered 
service.  Multiplicity  invariants  are  specified  in  the  Domain  Model  (e.g.,  an  Engagement  may  have  between  0  and  X  weapons), 
so  they  should  not  be  specified  here.  Note  that  when  a  service  terminates  due  to  exceptional  behavior,  part  of  the  component's 
exception  handling  should  insure  that  the  invariant  states  are  still  true.  If  they  are  not  true,  the  component  should  restore  the 
invariant  states.  If  this  is  not  done,  the  component  will  be  in  an  unstable  state. 

Exceptions: 

NameOf Exception  •  What  happens  if  component  initialized  when  preconditions  are  not  met. 

•  What  happens  if  preconditions  are  met,  but  postconditions  cannot  be  met. 

•  What  happens  if  an  invariant  fails  to  be  maintained. 

•  Notes 

o  An  exception  is  only  handled  if  explicitly  stated,  otherwise  component  behavior  is  undefined, 
o  "Exception"  means  behavior  in  the  event  of  contract  violation.  It  is  not  meant  to  imply  that  a  C++ 
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How  This  Technique  Addresses  Openness 

•  Provides  well-defined  interfaces  using  open  paradigms 

-DbC  and  UML  Domain  Modeling 

•  Generated  using  open  tools 

•  Output  is  readable  in  any  HTML  browser 
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EXPERIENCE.  RESULTS. 


What  the  Component  Spec  Provides 

•  Software  Architect 

-Defines  a  component’s  role  in  overall  architecture 
-Facilitates  component  reuse 

•  Software  Developer 

-Defines  implementation  constraints 
-Describes  exceptional  behavior 

•  System  Engineer 

-Facilitates  understanding  of  the  component’s  role  in  fulfilling 
requirements 

•  Component  Test  Engineers 

-  Provides  basis  for  writing  component  level  tests 
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EXPERIENCE.  RESULTS. 


Level  of  Effort  for  Sample  Component 

•  Task  requires  domain  knowledge 

-  Does  not  need  to  be  expert ,  but  does  need  access  to  an  expert 

•  Documented  40  services 

-28  trivial,  e.g.,  getters/setters 
-12  non-trivial 

•  3.5  staff  weeks 
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For  More  Information 


Kenneth  Klein 
kklein1@csc.com 
856-252-2359 

Joanis  Ploumitsakos 
jploumit@csc.com 
856-252-2091 
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On  Board  Inert  Gas  Generation  System 

(OBIGGS) 


•  Today  I’ll  cover: 

-  OBIGGS  project 

-  The  state  of  Systems  Engineering 

-  SE  implementation  on  project 

-  Project  results 
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MR  FORCE 


o  MR  FORCE 


U.S.AIR  FORCE 


U.S.AIR  FORCE  BSP* 


U.S.AIR  FORCE 

i3a 
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Dedicated  Personnel 


Team  Co-located  Facilities 


rJ o w  iho  Tesjj/j  was*  Prepared  io  Work 
To  cipher  ]n  Addjfesslmi  iho 


Engineering 


Production 


Field 

Services 


Training 


OBIGGS  II 
DIRECTOR 


Systems 

Engineering 


Field 


Flight 


Services 


Test 


Supplier 

Management 


NUMBER  OF  REMOVALS 


fBzini  ArJ3iJy^j£>  D'j  Dniri  io  Jd^mny 
j^osslbJej  rioui  CS1U^8£> 


Pareto  Analysis 


4  main  problem  components  were 
focus  of  initial  improvement  attempts 
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OBIGGS  COMPRESSOR  FAILURE  MODES 
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Components  failure 


5H 


Create  New 
Frontiers 


Profitably 

Expand 

Markets 

Operational 

Efficiency 

Customer 

Solutions 


Leverage  to 
Emerging 
Opportunities 


Run  Healthy 
Business 


_ 


Achieve  aggressive, 
sustainable  improvements 
to  safety,  quality, 
schedule  and  cost  ■ 

Strengthen  stakeholder  . 
relationships 

Relentlessly  improve  and 
integrate  processes  i 


Aggressively  pursue  a 
sustainable  competitive 
advantage 

Capture  additional 
C-17  business  (C-17, 
BC-17X,  International) 

Launch  C-17A+ 

Capture  Performance 
Improvement  contracts 

Expand  alliances  and 
partnerships 


■  Create  Agile  Logistics 
Mobility  and  Systems 
Solutions 

■  Create  Next  Generation 
Airlift/Support 

■  Create  Network-Centric 
Capability  Integration 

■  Accelerate  Technology 
Integration 

Our  Vision; 

People  Working  Together 

to  Provide  the  World's  First 
Choice  tor  Global  Airlift 
and  Mobility  Solutions 


Time 
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Customer 
Work  Force 
Suppliers 
Community 
Shareholders 


Community 

Shareholders 


Affeoiad 

pBnonminoB  jWe^u/e^  and  Biraianiaa 


Value 
Creation 

Profitably 
Expand 
Markets 


Operational 

Efficiency 

Customel 
Solution! 


Run  Healthy 
Business 


Achieve  aggressive, 
sustainable  improvements 
to  safety,  quality, 
schedufe  and  cost 

Strengthen  stakeholder 
relationships 

Relentlessly  improve  and 
integrate  processes 


:eate  New 
itiers 


Create  Agile  Lo 
Mobility  and  Syste 


olutions 


xt  Generation 


entric 


Our  Vision: 

People  Working  Together 
to  Provide  the  World’s  l-irst 
Choice  tor  Global  Airlift 
end  Mobility  Solutions 


Time 
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Stakeholders 


Internal 


Engineering 


Production 


Supplier  Management 


Support  Systems 


Training 
Field  Services 


Flight  Test 


External 


Pilots 


Maintainers 


Customer  Engineering 


How  Affected  Stakeholders  were 
Identified 

•  internal  stakeholders  identified  via  project 
management  process  at  kick-off  meeting 

•  External  customer  stakeholders  identified 
by  Boeing  Field  Services  and  USAF 
engineering  customers 

•  External  supplier  stakeholders  identified 
through  competitive  bid  process 


\ 


Suppliers 
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rJ 0 'j!/  ifjy 


MonthW 


Ba\ance 


amount 


How  ihs  Tori/jj  woo  Proporod  io  Work 
Topiorhor  jo  Addrooojj-jci  ihs  Projoci 


Training  Class 

Benefit 

System  Engineering  Workshop 

Requirements  definition 

Model  Based  Definition 

Eliminate  2-D  drawings 

Earned  Value  Management 

Performance  and  Cost  control 

Integrated  Performance  and  Scheduling 

Schedule  adherence 

Employee  Involvement 

Address  barriers  as  a  team 

Accelerated  Improvement  Workshops 

Tool  use  for  root  cause  analysis 
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Project  Team 
Stand-Up 


Action  item  review 


OCCURRENCE 

ATTENDEES 

Daily 

Internal  -  Supplier  Management, 
Systems  Engineering,  Project 
Management 

Weekly 

Customer,  Project  management 

Open  communication  was  emphasized  and  key 

to  project  success! 


Technical  Interchange 

Bi-monthly 
in  person 

Customer,  Project  management 

Internal  project  review 

Bi-monthly 

Boeing  executive  leadership 

Program  review 

Bi-Monthly  video 
conference 

Boeing  and  customer  executive 
leadership 
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Brainstorming 


Possible 

Solutions 
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Performance 


T23i;j]  ArjsjJysls  ol  Di ii3i  io  DavaJop 

PosslbJy  boJuiJLm^ 


Performance 


Reliability 


Copyright  ©  2007  The  Boeing  Company.  All  Rights  Reserved 


T23i;j]  ArjsiJysls  ol  Di ii3i  io  DavaJop 
Pa^jbJy  boJuiJLm^ 


Performance 


Reliability 


In 3  r^sjui  -De^jcJbcJ  ic)  U£>^  Jjtj 
BbMMIU cl  ih^.  rJ;J3ii  SoJuilD/J 


Design  Requirements 

5 

3 

1 

1.  Supports  tank  volume  of  cu  ft 

Supports  > 

Supports  < 

2.  Maintain  tank  and  vent  system  inert 
through  all  mission  profiles 

Tanks  and  vent  inert 
through  all  profiles 

Tanks  inert  through  all  profiles, 
vents  most 

Tanks  and  vents  inert  through 
most  profiles 

^  'Trxfcil  £»nmn£»  xxrifVnn  lumte 

<mo/„ 

^  %  1 

4.  Initialization  time  <  |  min. 

t  <  |  min. 

|  min.<  t  <  min. 

min.  <  t  |) 

5.  Mean-Time  Between  Maintenance, 
corrective 

MTBMc  >  hrs 

hrs  <  MTBMc  <  hrs 

MTBMc  <  hrs 

o.  Lite  Lycie  los^™ 

LCC  <  1)0%  oi  current 

yuu/o  ot  current  <  lll  <  current 

UJC  >  Current 

7.  No  increase  in  pilot  workload 

Decrease  in  workload 

Same  workload 

Slight  increase  in  workload 

10.  Qualified  components 

Qualified 

Partially  qualified 

Not  qualified 

1 1 .  Fuel  tank  pressures 

Meets  pressure  settings 

Doesn't  meet  pressure  settings 

12.  Single  ASM  failure  does  not  limit 
mission  capability 

All  missions  possible 

95%  of  missions  still  possible 

90%  of  missions  still  possible 

13.  Detect  individual  LRU  failures 

LRUs  identified  and 
isolated  by  BIT 

Failures  identified,  but  fault  tree 
required  for  isolation 

Periodic  ops  checks  and 
isolation  required 

14.  Capable  of  inert  fpm  descent 

with  any  single  failure 

fpm  possible  with 
all  single  failure  types 

fpm  possible  with  all 
except  |  failure  types 

fpm  possible  with  all 
except  >  |  failure  types 

15.  No  two  failures  cause  critical 
structural  failure  or  prevent  recovery 

No  critical  double 
failures 

Critical  double  failures  exist 

16.  No  Real  Hazard  I>1 1 

All  RHIs  <  8 

8  <  RHIs  <  1 1 

Some  RHIs  >  1 1 

17.  Current  cockpit  philosophy 

Integrated 

Pseudo  Integrated 

Not  integrated 

1 8 .  Capability  of  retrofit 

Easy  retrofit 

Hard  to  retrofit 

Can't  retrofit 

20.  General  design  practices 

Design  standards 
followed  in  all  areas 

Design  standards  followed  in 
most  areas 

Design  standards  followed  in 
some  areas 

21.  Production  Cost  Savings 

cs  >  $Hk 

$Hk  <  cs  <  $Hk 

cs  <  $Hk 

bJaibod^  31/jcJ  TddJs  \Jobl\  by  iho  'toiun  to 
SsJ^C'i  iho  rimiJ  BoJiniorJ^s 


Possible 

Solutions 


m 
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Assembled 
Stakeholder  Team 


Sizing 


Performance 


Reliability 


Cost 


Design  Requirements 

5 

3 

l 

1 .  Supports  tank  volume  of  cu  ft 

Supports  >  H 

Supports  <  H 

2.  Maintain  tank  and  vent  system  inert 
through  all  mission  profiles 

Tanks  and  vent  inert 
through  all  profiles 

Tanks  inert  through  all  profiles, 

Tanks  and  vents  inert  through 
most  profiles 

3.  Total  engine  flow  within  limits 

<■% 

>■% 

4.  Initialization  time  <  |  min. 

V  min.<  t  <^|min. 

5.  Mean-Time  Between  Maintenance, 

MTBMc  >  H  hrs 

■  hrs  <  MTBMc  <  ■  hrs 

MTBMc  <■  hrs 

6.  Life  Cycle  Costs 

LCC  <  90%  of  current 

90%  of  current  <  LCC  <  current 

LCC  >  Current 

7.  No  increase  in  pilot  workload 

Decrease  in  workload 

Same  workload 

Slight  increase  in  workload 

10.  Qualified  components 

Qualified 

Partially  qualified 

Not  qualified 

11.  Fuel  tank  pressures 

Meets  pressure  settings 

Doesn't  meet  pressure  settings 

12.  Single  ASM  failure  does  not  limit 
mission  capability 

All  missions  possible 

95%  of  missions  still  possible 

90%  of  missions  still  possible 

13.  Detect  individual  LRU  failures 

LRUs  identified  and 
isolated  by  BIT 

Failures  identified,  but  fault  tree 
required  for  isolation 

Periodic  ops  checks  and 
isolation  required 

14.  Capable  of  inert  fpm  descent 

with  any  single  failure 

Bl  fpm  possible  with 
all  single  failure  types 

|B  fpm  possible  with  all 
except  |  failure  types 

|B  fpm  possible  with  all 
except  >  |  failure  types 

15.  No  two  failures  cause  critical 
structural  failure  or  prevent  recovery 

No  critical  double 
failures 

Critical  double  failures  exist 

16.  No  Real  Hazard  I>11 

All  RHIs  <  8 

8  ^  RHIs  <  1 1 

Some  RHIs  >11 

17.  Current  cockpit  philosophy 

Integrated 

Pseudo  Integrated 

Not  integrated 

18.  Capability  of  retrofit 

Easy  retrofit 

Hard  to  retrofit 

Can't  retrofit 

20.  General  design  practices 

Design  standards 
followed  in  all  areas 

Design  standards  followed  in 

Design  standards  followed  in 
some  areas 

21.  Production  Cost  Savings 

cs>$Hk 

$Hk<cs<$Hk 

cs  <  $Hk 

Performed 
Trade  Study 


Presented  Analysis 


Final 
Solution1 
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Operations 


Suppliers 


Air  Force 
Engineering 


Logistics 

Support 


TRADE  STUDY 


Production 


Maintainers 


Engineering 
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Flight  Crews 
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WEEP 


VENT 


OBIGGS  2 
SYSTEM 
,EFT  HAND 
SIDE 


NEA  SUPPLY 
HIGH  POINT 


NEA  VENT  AND 
SWEEP  VALVES 


TANK  2 

PENETRATION 


REAR  SPAR 


BILGE  CRAWLER 


NEA  SUPPLY 


NEA  TO  TANK1 


"AIR  CROSSOVER 


BLEED  AIR  INLET 


AIR  FILTER 


■  SIGGS  HEAT  EXCHANGER 


NEA  MANIFOLD 


AIR  MANIFOLD 


ECS  PACK 


COMPRESSOR 


ASM  SUPPLY 


OVERBOARD  EXHAUST- 
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Risk  Management 


TASMO 


Tmijj]  w 


Jjjjpiemem  its  £3  uJ  ini  cm 


Project 

Task 

Plan 


Integrated  Master 
Schedule 
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PROJECT/PROGRAM 

C-17 


John  C.  Hsu  (982-6559) 


WBS  LEVEL 

1  2  3  4  5  6  7  8 


3.2,1 


WORK  BREAKDOWN  STRUCTURE 
DICTIONARY 

OBIGGS  REDESIGN 


TOTAL  PROGRAM 


DATE:  12/02/99 
REVISION:  1 


S.O  W  NO./PARA. 


ELEMENT  TITLE 


Environmental  Systems 


ELEMENT  DESCRIPTION 

3.2.1  Perform  trade  study 

3  2.1.1  Define  customer  requirements  (Use  actual) 

3.2. 1.2  Define  system  requirements  (Use  actual) 

3.2.1.3  Identify  candidate  systems  (Use  actual) 

3.2.1.4  Define  QFD  relationships  (Use  actual) 

3.2. 1.6  Define  customer  werghis  (Use  actual) 

1.6  Establish  scoring  criteria  (Use  actual) 

3.2. 1.7  Rank  candidate  systems  (Use  actual) 

3  11 .15.1  Update/Monitor  customer,  systems,  design  and  derived  requirements 
3.11 .15.2  Sub-systems/components  trade  study 

EFFORT  REQUIRED 


1  *34  »678 


REVISION  APPROVAL: 


ASSOCIATED  LOWER  LEVEL  ELEMENTS 
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j-Jow  Stakeholder  Buy-In  Was  Bnsursd 


□I 


Stakeholders 

Plan  to  Ensure  Buy-in: 

Validated  By: 

Engineering 

Developing  own  implementation  plans.  Reported 
progress  to  them  regularly. 

Dedicated  support  to  the  project.  Commitment  to 
plan  evident  during  regular  status  reviews. 

Production 

Early  involvement  for  development  of  installation 
plans.  Collocated  engineers  on  first  assembly. 

Full  scale  mockups  of  large  parts. 

Requests  for  manufacturing  features  on  designs. 
Strong  participation  in  mockup  trial  installations. 
Positive  feedback  during  first  installations. 

Supplier 

Management 

Early  close  coordination  with  engineering, 
participation  in  drawing  release  reviews 

Strong  participation.  Provided  part-by-part  status 
weekly.  Aggressive  resolution  of  issues. 

Support 

Systems 

Development  of  own  performance  metrics  and 
reporting  progress  to  stakeholders 

Enthusiastic  participation  in  design  reviews.  Early 
coordination  of  validation  impacts  with  customer. 

Training 

Early  coordination  with  engineering  aided  course 
development 

Early  development  of  plan,  communication  with 
project  team  and  customer 

Field 

Services 

Early  visibility  from  design  reviews.  Aided 
planning  of  future  customer  support 

Initiative  in  learning  the  system  prior  to  first  delivery 

Flight  Test 

Full  time  interaction  with  design  team,  from 
development  through  test  flights 

Outstanding  management  of  installation  of 
instrumentation  in  production.  Close  coordination 
with  engineering  when  developing  test  plans. 

Pilots 

Dramatic  potential  improvement  of  inerting 
system 

Affirmation  during  base  visits 

Maintainers 

Design  reviews  at  bases  prior  to  implementation. 
Participation  in  mockup  installation. 

Enthusiastic  participation  at  bases  during  reviews, 
mockup  installation,  follow-up  communication 

Customer 

Engineering 

Involvement  in  project  selection.  Frequent, 
regular  communication.  Full  system  lab  test. 

Strong  support  for  project.  Teamwork  in  decisions 
addressing  challenges,  regular  communication. 

Suppliers 

Frequent  communication,  design  reviews,-  they 
were  team  members 

Strong  participation  in  developing  design  solutions. 
Commitment  to  schedule  needs. 
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Type 

How  Identified 

How  Addressed 

Customer  reluctance  to  fund 
project  due  to  high  cost 

Customer  feedback  during 
negotiations 

Detailed  estimates,  competitive 
pricing  &  life  cycle  cost  analysis 

Supplier  not  willing  to  control 
interfaces  to  requested 
tolerances 

Interface  Key  Characteristic 
reviews 

Negotiated  compromise  during 
weekly  supplier  coordination 
meetings 

Production  schedule  impact 
from  late  parts 

Feedback  from  production 
stakeholder  on  team 

Established  agreed-to  lead 
times  for  parts 

Production  schedule  impact 
from  learning  curve 

Feedback  from  production 
stakeholder  on  team 

Fit  checks,  dedicated 
engineering  support 

Production  concern  about 
part  damage  on  installation 

Feedback  from  production 
stakeholder  on  team 

Assembly  simulation  and 
created  protective  covers 

Cluttered  production  work 
space 

Lean  initiatives  coordination 
meetings  with  Production 

Created  point-of-use  carts  to 
transport  selected  parts 

Flight  test  airplane  out  of 
service  too  long 

Customer  feedback  during 
flight  test  planning 

Installed  instrumentation  in 
production 

Resistance  to  Model  Based 
Definition  from  QA 

QA  feedback  at  first  article 
inspection 

Generated  2D  inspection 
sheets  from  3D  models 
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Lean  initiatives  coordination 
meetings  with  Production 
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Flight  test  airplane  out  of 
service  too  long 

Customer  feedback 
during  flight  test  planning 
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Definition  from  QA 

QA  feedback  at  first  article 
inspection 

Generated  2D  inspection 
sheets  from  3D  models 
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OBIGGS  II  vs.  OBIGGS  I  RELIABILITY  DEMO  RESULTS 


(US  Air  Force  Photo) 


Tangible  Benefits 


Achieved  7400%  Increase  in 
system  reliability  vs.  1100% 


(US  Air  Force  Photo) 


Reduced  Initialization  Time 
by  a  factor  of  1 1  vs.  5 


Reduced  weight  by  517  lbs 
vs.  475  lbs.  allowing  for 
increased  cargo  capability 


20%  system  and  3:1  life  cycle 
cost  savings  as  predicted 
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Process 

Improvements 

•  Four  different 


Reduced  Initialization  Time 
•  Improved  by  a  factor  of  1 1 


/Gate  New 
Frontiers 


Value 

Creation 


Leverage  t 
EmeraK" 


Create  Agile  Logistics 
Mobility  and  Systems 
Solutions 


Profitably 

Expand 

Markets 


Create  Next  Generation 
Airlift/Support 


Create  Network-Centric 
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■  Achieve  aggressive, 
sustainabfe 

improvements  to  safety 
quality,  schedule  and 
cost 


- - - CJL1L  USLULL 

People  Working  Together 
to  Provide  the  World's  First 
Choice  for  Global  Airlift 
and  Mobility  Solutions 


rmance 

contracts 

^^nd 


Strengthen  stakeholder 
relationships 


relationships 

Relentlessly  improve  and 
integrate  processes 
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Mission  Accomplished! 


The  Joint  Partnership 
between  Program 
Management  & 
Systems  Engineering 
on  Support  System 
Program 


Samuel  Son 

and 

Mark  Keller 


Topics  of  Discussion 


Overview 


The  Partnership  Umbrella 

-  Program  Management  (PM) 

-  Systems  Engineering  (SE) 

-  Interrelationships  Between  PM  and  SE 


•  Systems  Engineering  Process  Stabilization  &  Enhancement  w 

-  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

-  Project  Performance  Assessment  &  Review  Process 

-  Program  Management  Best  Practices  (PMBP) 

-  Systems  Engineering  Best  Practices  (SEBP) 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Total  System  Support  Responsibility  (TSSR) 

•  Conclusion 


10/26/2007 
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Overview 


Support  System  Program 

-  Provide  warfighter  sustainment  that  guarantees  readiness,  aircraft  availability,  and 
affordability 

Program  Management 

-  Management  of  key  program  items,  such  as  costs,  timely  delivery,  people,  quality,  and 
risks 

Systems  Engineering 

-  Ensures  common  application  of  Systems  Engineering  processes,  implementation,  and 
execution  to  facilitate  program  and  mission  success 


Program  Management  and  Systems  Engineering,  along  with  government  and 
industry  best  practices,  become  interdependent  to  successfully  monitor,  measure, 
manage  and  execute  Support  System  Integration  activities 

=  Warfighter  Sustainment 
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Topics  of  Discussion 


Overview 


The  Partnership  Umbrella 

-  Program  Management  (PM) 

-  Systems  Engineering  (SE) 

-  Interrelationship  Between  PM  and  SE 


•  Systems  Engineering  Process  Stabilization  &  Enhancement 

-  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

-  Project  Performance  Assessment  &  Review  Process 

-  Program  Management  Best  Practices  (PMBP) 

-  Systems  Engineering  Best  Practices  (SEBP) 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Total  System  Support  Responsibility  (TSSR) 

•  Conclusion 
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The  Partnership  Umbrella:  Program  Management 


TPMs  KPPs  Program 
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The  Partnership  Umbrella:  Systems  Engineering 
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The  Partnership  Umbrella:  Interrelationships 


Program  Management 


Systems  Engineering 
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Promoting  improved  implementation  of  SE  and  PM  Best  Practices 
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The  Partnership  Umbrella:  Interrelationships 


Program  Management 


Systems  Engineering 
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Topics  of  Discussion 


Overview 


The  Partnership  Umbrella 

-  Program  Management  (PM) 

-  Systems  Engineering  (SE) 

-  Interrelationships  Between  PM  and  SE 


•  Systems  Engineering  Process  Stabilization  &  Enhancement 

-  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

-  Project  Performance  Assessment  &  Review  Process 

-  Program  Management  Best  Practices  (PMBP) 

-  Systems  Engineering  Best  Practices  (SEBP) 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Total  System  Support  Responsibility  (TSSR) 

•  Conclusion 
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SE  Process  Stabilization  &  Enhancement  (examples) 


•  Systems  Engineering  Supporting  Program  Management 

-  Provide  Systems  Engineering  Processes 

-  Perform  Assessments  &  Best  Practices 

•  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

•  Project  Performance  Assessment  &  Review  Process 

•  Program  Management  Best  Practices  (PMBP) 

•  Systems  Engineering  Best  Practices  (SEBP) 


•  Assessments  &  Best  Practices  provide  total  visibility  on  strengths  and 
weaknesses  in  Systems  Engineering  as  well  as  progress  of  improvement  efforts 
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Contractor  Performance  Assessment  Reporting  (CPAR) 


•  Objectives: 

-  Ensure  that  accurate  data  on  contractor  performance  is  current  and  available  for  use 
in  source  selections 

-  Consistently  provide  quality,  on-time  products  and  services  that  conform  to  contractual 
requirements 

-  Effectively  communicate  contractor  strengths  and  weaknesses  to  source  selection 
officials 

•  Systems  Engineering  Supporting  Program  Management: 

-  Use  Award  Fee  Rating  Criteria 

-  Review  Customer’s  AFAST  Database 

-  Review  Award  Fee  Review  Charts 

-  Review  Project  Integration  Weekly  Reports 

-  Field  Service  Weekly  Reports 
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Project  Performance  Assessment  and  Review  Process 


•  Objectives: 

-  To  rate,  assess,  and  report  project  performance  to  management  and  the  customer 

•  Systems  Engineering  Supporting  Program  Management: 

-  Review  Technical  Performance  Measurement 

-  Review  Systems  Engineering  Compliance 

•  Requirements,  Risk,  Verification,  Formal  Review,  and  Critical  Action  ltem(s) 


•  Support  Systems  Supporting  Program  management: 

-  Review  Support  Systems 

•Tech  Orders,  Support  Equipment,  Spares,  and  Repair  of  Repairable 

-  Review  Trainings 

•  Maintenance  ‘Type-1’  Training 

•  Retro  Training 
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Program  Management  Best  Practices  (PMBP) 


•  Objectives: 

-  To  achieve  successful  program  development,  implementation  and  support  based  on 
an  integrated  set  of  Program  Management  Best  Practices 

•  Systems  Engineering  Supporting  Program  Management: 

-  Review  maturity  level  for  program  execution  &  control 

-  Use  program  execution  &  control  best  practice  criteria 

•  Allocation  and  traceability  of  program  requirements 

•  Identification  of  Program-level  KPPs 

•  Allocation  and  traceability  of  TPMs 


Business 

Business  Plan 

Program  Execution  & 

Organization 

Proposition 

Control 

Program 

Supplier 

Risk,  Issue  & 

Help  Needed  & 

Communication 

Integration 

Opportunities 

Independent 

Review 

10/26/2007 
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•  Objectives: 

-  Strengthen  Systems  Engineering 

-  Maintain  the  Capability  Maturity  Model  Integration  (CMMI)  Level  5 

•  Systems  Engineering  Supporting  Program  Management 

-  Develop  Systems  Engineering  Best  Practices  Self  Assessment  Plan 

-  Review  overall  attributes  associated  with  each  of  the  Best  Practices 

-  Develop  Systems  Engineering  Management  Plan  to  include  the  Support  System 

-  Improve  training  materials 

•  Requirements  Management 


-  Provide  Systems  Engineering  training  to  Project 
Managers 


•  Risk  Management 
•Technical  Performance  Measures 
•Trade  Studies 

•  Verification  &  Validation 


10/26/2007 
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Topics  of  Discussion 


Overview 


The  Partnership  Umbrella 

-  Program  Management  (PM) 

-  Systems  Engineering  (SE) 

-  Interrelationships  Between  PM  and  SE 


•  Systems  Engineering  Process  Stabilization  &  Enhancement 

-  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

-  Project  Performance  Assessment  &  Review  Process 

-  Program  Management  Best  Practices  (PMBP) 

-  Systems  Engineering  Best  Practices  (SEBP) 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Total  System  Support  Responsibility  (TSSR) 

•  Conclusion 
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Users  of  the  SE  Process  at  Multiple  Organization  Levels 


LU 


O 

< 


LU 

</> 


r-  Enterprise  Level 


>-  Program  Level 

>-  Integrated  Product  Team  Level 

>-  Project  Level 

Image  Source:  University  of  Toronto  Magazine 
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BOEING 


Support  Systems 


Aircraft  Availability 
Flying  Hours  Achievable 
Mission  Capability 
Maintenance  Scheduling 
Issue  Effectiveness 
Customer  Satisfaction 


Program-level  KPPs 


IPT-level 


Integrated 

Support 

Technical 

Retrofit  & 

System 

Support 

Analysis 

Training 

Field 

Planning  & 
Management 

Publications 

Modification 

System 

Services 

Supply  1 1  Support 
Support  Equipment 
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Six  Program-level  KPPs  for  Support  Systems 


AIRCRAFT  AVAILABILITY  AND  CUSTOMER  SATISFACTION  ARE  PARAMOUNT 


10/26/2007 
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Support  Systems  -  Integrated  Product  Teams 


Supply 

Support 


Field 

Services 


Training 

Systems 


Integrated 
Support 
Planning  & 
Management 


Technical 

Publications 


Retrofit  & 
Modification 


System 

Support 

Analysis 


Support 

Equipment 


Customer  Satisfaction 
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Topics  of  Discussion 


Overview 


The  Partnership  Umbrella 

-  Program  Management  (PM) 

-  Systems  Engineering  (SE) 

-  Interrelationships  Between  PM  and  SE 


•  Systems  Engineering  Process  Stabilization  &  Enhancement 

-  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

-  Project  Performance  Assessment  &  Review  Process 

-  Program  Management  Best  Practices  (PMBP) 

-  Systems  Engineering  Best  Practices  (SEBP) 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Total  System  Support  Responsibility  (TSSR) 

•  Conclusion 


10/26/2007 


20 


Total  System  Support  Responsibility  (TSSR) 


•  What  is  TSSR?: 

-  A  program  built  on  the  performance-based  approach  that  uses  the  combination  of  best 

of  government  and  industry  practices  to  provide  support  program  affordability  and 
improved  aircraft  availability 

•  Benefits: 

-  Provides  the  customer  with  an  affordable  and  optimum  sustainment  solution:  as 
single  source  that  guarantees  support,  readiness,  availability,  24/7  customer 
service,  and  equates  to  a  more  efficient,  effective,  and 

consistent  support  program 

-  Ability  to  move  technical  data  into  the  field  faster 

-  Directing  maintenance  to  each  individual  aircraft’s 
weaknesses  before  malfunctions  occur 

-  Balances  heavy  maintenance  workload  and  ensures 
reserve  capacity 
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Total  System  Support  Responsibility  (TSSR)  Cont’ 


Customer  Satisfaction 


•  Keep  services  affordable 

•  Increase  fleet  availability 

•  Reduce  cycle  time 


10/26/2007 
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Topics  of  Discussion 


Overview 


The  Partnership  Umbrella 

-  Program  Management  (PM) 

-  Systems  Engineering  (SE) 

-  Interrelationships  Between  PM  and  SE 


•  Systems  Engineering  Process  Stabilization  &  Enhancement 

-  Contractor  Performance  Assessment  Reporting  (CPAR)  Review 

-  Project  Performance  Assessment  &  Review  Process 

-  Program  Management  Best  Practices  (PMBP) 

-  Systems  Engineering  Best  Practices  (SEBP) 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Total  System  Support  Responsibility  (TSSR) 

•  Conclusion 
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Conclusion 


•  The  synergistic  partnership  between  Program  Management  and  Systems 
Engineering  on  Support  System  Program  is  an  essential  enabler: 


-  To  keep  services  affordable 

-  To  increase  fleet  availability 

-  To  improve  effectiveness  and  reduce  cycle  time 

•  Benefits  to  the  weapon  system 

-  More  responsive  to  mission  demands 

-  Higher  quality  services  &  products 

-  On  time  deliveries  -  reduced  depot  time 

-  Increased  weapon  system  availability 
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Conclusion 


•  Benefits  to  the  Customer 

-  Reduced  cycle  times 

-  Easier  to  execute  purchasing  arrangements 

-  Fewer  transaction 

-  Lower  support  costs 

•  Benefits  to  suppliers 

-  More  predictable,  longer  term  business 

-  Strategic,  focused  relationships 

-  Fewer,  higher-value  contracts 

-  Lower  overhead  costs 


10/26/2007 
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Contact  Information 


•  Samuel  Son,  Systems  Engineer 

-  Phone:  (562)  982-2209 

-  Email:  samuel.k.son(a>boeinq.com 

•  Mark  Keller,  Systems  Engineer 

-  Phone:  (562)  593-8450 

-  Email:  mark.keller@boeinq.com 


Thank  you! 
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NORTHROP  GRUMMAN 


Defining  the  future 


A  Day  in  the 
Life  of  a 
Verification 
Requirement 


Systems  Engineering  Conference 
National  Defense  Industries  Association 


Oct.  22-25,  2007 

Stephen  Scukanec 

Test  and  Evaluation 

James  van  Gaasbeek 

Requirements  Development,  Management  and  Verification 
Northrop  Grumman  Corporation 

Approved  for  Public  Release,  Control  No.  07-097,  dtd.  10-3-07 


Agenda 

■  Why  Verification 

■  Overall  Process 

■  Verification  Cross-Reference  Matrix 

■  Verification  Attributes 

■  Requirement  Samples 

■  Verification  Plans 

■  Benefits 

■  Summary  /  Conclusions 

■  Abstract 

■  Author  Biographies 


NDIA  SED  Conference,  Wednesday,  24  October  2007,  Track  1,  Paper  5536 

Approved  for  Public  Release,  Control  No.  07-097,  dtd.  10-3-07 


NORTHROP  GRUMMAN 


Verification  Requirements  -  What  Are 
They  And  Why  Do  We  Need  Them? 

fA  ■  Verification  requirements  specify  the  verification 
^  events  needed  to  prove  the  satisfaction  of  the  product 
k  \  requirements  and  help  to  define  the  verification 
process  and  environment 

!■  Verification  requirements  are  necessary  for  at  least 
two  reasons: 

■  Existence  of  verification  requirements  demonstrates  verifiability  of 
product  requirements 

■  Agreed-to  verification  requirements  define  the  verification  program 
by  which  the  contractor  shows  that  the  product  is  what  the  customer 
needed 


NORTHROP  GRUMMAN 
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A  Day  in  the  Life  of  a  Verification 
Requirement 


Product  Requirements 


Design  Specification  Section  4  Charactistics 


m - 

[ 

IPT  - 

IPT  - 

[ 

IPT  - 

■ 

=  ■  =  ■  =  ■  =  ■  =  ■  = 

Establish 

Establish 

Establish 

Develop 

■  Develop  p[etaflbGl : 

Proof  of  Design 

Certification  /  SOF 

Acceptance 

Verification  Matrix 

: : :  VdrificaiticuT  : : : 

Verification 

Verification 

Verification 

(VCRI/VCRM) 

; :  Rqqirirqmphts  ; : 

Statements  (A) 

Requirements  (B) 

Requirements  (C) 

(D)  _ 

-  IPT/V6 

-  IPT/V&V 

IPT/V&V 

IPT/V&V 

IPT/V&V 

[wy- 

Generate 

Perform  Data 

Verification 

Analysis 

Method  Report 

(K) 

Generate 
Verification  Data 
Package  and 
Submit  for  Approval 
(L) 


ISJ 


Report 


Archive  Data 
Package  With 
Configuration 
Management 


YesS 


Submit  Package 
To  Customer 
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Verification  events  satisfy 
the  verification 
requirements,  NOT  the 
product  requirements. 

Product  requirements  are 
never  complete  until  the 
associated  verification 
requirements  are 
completed 

The  culmination  of  the 
verification  activity  of  the 
design  requirements 
results  in  a  verified 
product. 


DD-250 


NORTHROP  GRUMMAN 


Start  with  Product  Requirements _ 

■  The  verification  process  begins  with  authenticated 
ri  product  requirements 

*1  ■  Examples 

k  ■  PR-1  :LRU  markings 

■  The  product  line-replaceable  units  shall  be  marked  in  accordance 
with  MIL-STD-130M. 

■  ■  Pr-2:  operational  availability 

-  The  product  shall  have  an  operational  availability  (A0)  of  97.5%  at 
I  IOC. 

■  -  Pr-3:  Iru  accessibility 

■  -  Each  product  line-replaceable  unit  shall  be  able  to  be  removed 

and  replaced  without  removing  any  other  item  or  displacing  any 
cables. 

■  Pr-4:recovery  force  communication  -  nominal 

■  The  product  shall  provide  a  communications  system  capable  of 
communicating  with  the  ground  command. 
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Create  Verification  Cross-Reference 
Matrix 


I 


Design 
Requirement 

V 


3.2.2.15.34  Recovery  Force  Communications 
The  communications  system  shall  provide  a  communications 
> system  capable  of  communicating  with  the  recovery  forces  pre- 
and  post-  landing 


Verification  Objective 


Perform  Integrated  System 
Test  of  the  communications 
system  capability  to  provide  a 
voice  communications  and 
[\J  beacon  with  recovery  forces 
J  X  pre  and  post  landing  within 
.,  ...  t.  >an  integrated  hardware  / 
Verification  /software  environment 


Design 


1/ 


Perform  a  demonstration  of 
the  communications  systems 
capability  to  provide  voice 
and  beacon  communications 
with  recovery  forces  pre  and 
post  landing  while  within  a 
representative  environment 
and  using  a  production 
equipment  configuration 


Pass  /  Fail  (Success  Criteria) 


Testing  will  show  that  the 
communications  system  can 
transmit  and  receive  audio  at 
frequencies  and  ranges 
(power)  represented  by 
standard  ground  recovery 
force  communications 
devices  as  defined  in  TBD 


Testing  will  show  the  ability 
for  the  communications 
systems  to  verbally 
communicate  with  the  on 
board  communication 
production  configuration 
equipment.  The 
demonstration  will  also  show 
beacon  tracking  within 
communication  ranges 
established  by  TBD. 


Verification  Cross  Reference  Matrix 


Traceability 

L 


Paragraph  # 

N/A 

1 

A 

M/S 

D 

T 

Y-2.2.15.34 

T3 

{  3.2.2.15.34 

T5 

SE  -  Translates  Operational 
Objectives  into  Design 
Requirements 

Design  -  Provides  assessment  of 
requirements  implementation 

Test  -  Provides  assessment  of 
requirements  verifiability 


SE  -  Provides  compliance  of  the 
design  requirement 

Test  /  Implementation  Group  - 

Ensures  Verification 
Implementation  Feasibility 

Advises  alternatives  to  support 
programmatics 

Assess  completeness 

Provides  verifiability  assessment 


SE  -  Verification  Allocation  and 
Traceability  Assurance 


Identifvinq  a  verification  method  is  necessary,  but  not  sufficient! 
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Verification  Requirement  Attributes 


Verification 

Requirements 


•Inspection 

•Analysis 

•Modeling  and 

Simulation 

•Demonstration 

•Test 


Verification  isn’t  ONLY  test! 


^Objective 

What  is  the  purpose  of  this  verification? 

ElMethod 

What  method  do  you  need  performed?  What 
are  the  verification  circumstances  (e.g., 
laboratory,  desk-top  analysis,  flight  test)? 

^Environment 

What  are  the  environmental  conditions  under 
which  the  item  will  be  verified? 

LdSpecial  Conditions  (if  necessary) 

Are  there  any  unique  conditions  (e.g.,  item 
configurations)  necessary  for  the  execution  of 
the  verification? 

BSuccess  Criteria 

What  results  are  to  expected? 
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Sample  Verification  Requirements  -  1 


A  ■  VR-11:  compliance  of  product  markings  shall  be 
‘’l  verified  by  examination  of  design  drawings  at  the  LRU 
1  supplier’s  location  prior  to  the  LRU  CDR.  The 
inspection  will  show  that  each  marking  on  the  LRU 
conforms  to  MIL-STD-130M. 


■  Vr-2a:  the  product  operational  availability  shall  be 
calculated  using  the  results  of  the  government- 
accredited  contractor-developed  reliability  and 
maintainability  analyses  performed  during  the  design 
in  conjunction  with  the  design  reference  missions 
documented  in  report  xxxx.  The  analysis  will  show 
that  the  product,  in  its  operational  environment, 
supported  with  its  support  equipment  and  personnel, 
across  all  missions,  will  have  an  operational 
availability  of  at  least  97.5%. 
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Sample  Verification  Requirements  -  2 

■  VR-3D:  Removal  and  replacement  of  all  Irus  shall  be 
A  demonstrated  on  the  aircraft  to  show  that  each  LRU 
'  :  can  be  removed  and  replaced  without  removing  any 
1  other  items  or  moving  any  cables. 

1“  ■  Vr-4d:  Perform  demonstration  to  provide  a 

communications  system  capable  of  communicating 
with  the  ground  command  team  while  in  a 
representative  environment  and  production 
configuration.  Demonstration  will  show  capability  to 
communicate  with  recovery  forces  at  TBD  distances  in 
the  TBD  terrain  environment. 
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Sample  Verification  Requirements  -  3 

■  VR-4T :  Prove  that  the  product’s  communications 
system  is  capable  of  communicating  with  the  ground 
command  team  by  performing  an  integrated  system 


test  within  an  integrated  hardware/software 

environment. 

Testing  will  show  that  the  product  can 

transmit  and  receive  audio  at  frequencies  represented 

by  standard  ground  recovery  forces  communications 

devices  defined  in  (TBD). 

Environment 


Note  -  there  are  no 
Special  Conditions 
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Verification  Requirements  Flow  and 
Traceability 

Specification 


PR-1 

PR-2 

PR-4 

PR-^ 

VR-11 

VR-2A 

VR-4D 

VR-5T 

Verification  Requirements  Appear 
in  the  Same  Specification  as  the 
Product  Requirements  to  be  Verified 


Design 

Requirements 


Verification 
Requirements 


Master  Verification  Plan 


Inspection  VR-11 
Analysis  VR-2A 
Modeling  and 
Simulation 
Demonstration 
VR-3D,  VR-4D 
Test 
VR-4T 


Product 

Requirement 

N/A 

Insp 

Analy 

M&S 

Demo 

Test 

Verification 

Requirement 

PR-1 

X 

VR-11 

PR-2 

X 

VR-2A 

PR-3 

X 

VR-3MS 

PR-4 

X 

VR-4D 

PR-5 

X 

X 

VR-5D 

VR-5T 

VerlfA 


rf/on 
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Create  Detailed  Verification  Requirements 
(Verification  Events) 


Convert  verification  statements  into 
detailed  verification  requirements 
(verification  events)  by  —— 


Master  Verification 


A  One  To  One  Relationship  Exists  Between 
the  Verification  Requirements  and  the  DVRs 


I For  each  verification  activity  identified  in  the 
verification  matrix,  a  detailed  description  of  the 
activity  including: 


< 


•Verification  configuration  &  its  relationship  to 

production  configuration 

•Associated  prerequisites 

•Constraints 

•Objectives 

•Procedures 

•Relevant  environmental  conditions 
•Pass/fail  criteria-  and  necessary  Data  Set, 
•Analysis  models,  if  applicable. 

•Sequence  if  applicable 

•Verification  Environment  (i.e.;  Lab,  Flight, 


^Production) 
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Master  Verification  Plan 


Master  Verification  Plan 


Inspection  VR-11 
Analysis  VR-2A 
Modeling  and 
Simulation 
Demonstration 
VR-3D,  VR-4D 
Test 
VR-4T 


III) 


Inspection 

Plans 

VR-11 


Analysis 

Plans 

VR-2A 


cc 


Modeling  / 
Simulation 
Plans 


Test/ 

Demo  Plans 

[|VR-3D,  VR-4D, 
VR-4T 
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Verification  Execution  Flow 


Method 

Organization 

Early  Verification  Benefits 

Inspection 

QA,  Manufacturing, 

Mission  Assurance 

■Inspection  Points  Identified 
■Tooling  Requirements  Identified 

Analysis 

Systems  Engineering 
Specialty  Engineering 

Design 

■Define  /  Build  /  Buy  /  Train  Analysis 

Prior  to  Need  Date 

■Accreditation  of  Analyses  Tools  Prior 
to  Need  Date 

Modeling  and 
Simulation 

Systems  Engineering 
Specialty  Engineering 

Design,  Operational 
Assessment 

■Define  /  Build  /  Buy  /  Train  Modeling 
and  Simulation  Tools  Prior  to  Need 

Date 

■Accreditation  of  Models  Prior  to  Need 
Date 

Demo  &  Test 

Ground  and  Flight  Test 
Facilities  Development 

■Laboratory  and  Lab  Software 
Requirements  Identified 

■Facilities  Requirements  Identified 

■Long  Lead  Test  Items  Identified 

CD' 
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Planning  for  Verification  Execution  and 
Product  Verification 


Revl 


Define  Verification 
Requirements  Early 
and  in  Detail  to 
Establish  the  Entire 
Verification  Effort 


0 


...  and  it  Costs 
Relatively  little  ... 


Rev  2 


Rev  3 


Long  Lead  Facilities 
Laboratory  Design 


Range  Coordination 


Design  Requirements 
Software 
Analysis  Tools 


Rev  4 


Requirerm 

ints 

Design 

B 

uild 

Verifies 

ion 

C 

Certification 


Discover  the  Verification 
Requirements  Late  and  Have 
Enormous  Rework  to  Establish 
the  Entire  Verification  Effort 


Early  Verification  Is  an  Effective  Cost  Avoidance  Approach 
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Summary  and  Conclusions 


~a  ■  The  verification  process  begins  with  authenticated  product 
^  requirements 

I  ■  Define  verification  requirements,  not  just  methods  -  the  VCRI  is 
'  the  last  thing  developed  in  the  specification 

■  Verification  requirements  must  state  the  objective,  method, 
environment,  and  expected  results.  There  may  also  be  special 
conditions. 

■  The  master  verification  plan  is  the  guidance  for  the  verification 
program 

I  ■  Verification  is  conducted  against  the  product  defined  by  the  title 
of  the  specification 

I  ■  Verification  program  benefits  are  not  limited  to  just  the  systems 
engineering  and  test  organizations 

I  ■  Define  the  verification  requirements  early  to  reduce  the  overall 
program  cost 
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Abstract 


One  measure  of  the  quality  of  a  product  requirement  is  that  it  be  verifiable.  Verifiability 
assessment  is  one  of  the  exit  criteria  for  the  Systems  Requirements  Review  and  is  necessary 
for  requirement  validity.  Nomination  of  one  or  more  verification  methods  (inspection, 
analysis,  modeling  and  simulation,  demonstration  or  test)  is  often  taken  as  the  sole  evidence 
of  verifiability.  A  completed  Verification  Cross  Reference  Matrix  is  frequently  considered  as 
the  final  verifiability  assessment  and  responsibility  for  the  remainder  of  the  verification  effort 
is  transferred  to  the  test  and  evaluation  and  other  implementing  communities  for  completion. 


■  Lessons  learned  from  many  Programs  have  shown  that  a  more  robust  application  of 

systems  engineering  should  include  the  requirements  engineers  (with  detailed  knowledge  of 
product  requirement  intent)  working  with  the  implementing  organizations  as  the  best 
combination  to  define  the  verification  requirements.  Such  definition  should  include 
statement  of  the  verification  objectives,  success  criteria  and  environment.  Including  this 
information  in  the  ’’Quality  Assurance”  section  of  the  requirements  document  allows  for  buy- 
in  by  the  customer  well  in  advance  of  implementing  the  verification  activities.  This 
information  is  used  by  verification  personnel  to  generate  one  or  more  verification  plans  and 
to  develop  the  detailed  verification  program.  Verification  requirements  are  planned  into 
verification  events  which  are  executed  using  the  proper  system  elements  and  environments. 
These  verification  requirements  are  key  to  establishing  long  lead  verification  facilities,  tools 
and  laboratories.  Early  definition  of  these  requirements  helps  prevent  facility  re-designs  and 
verification  re-plans  that  can  cause  expensive  delays.  Finally,  verification  data  analysis  is 
performed,  ana  the  information  compiled  into  verification  reports  certifying  system  product 
requirements  compliance.  This  robust  verification  approach  will  provide  proof  of 
requirements  satisfaction,  leading  to  systems  that  meet  the  customers'  needs  at  a  lower  life- 
cycle  cost. 


■  This  paper  describes  these  concepts  and  steps  in  detail  and  provides  examples  for  a  set  of 
generic  aircraft  requirements. 
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From  Science  to  Solutions 


Introduction 


•  Simulation  Based  Acquisition  (SBA)  programs  are  confronted  with 
two  paradigms  that  compete  for  program  level  focus  and 
resources. 

•  The  first  paradigm  requires  modeling  &  simulation  (M&S)  teams  to 
develop  simulation  environments  for  testing  in  order  to  “sell-off”  the 
SBA  program.  This  line  of  thought  invariably  demands  immediate 
attention  toward  developing  unique  simulation  based  event 
configurations  for  supporting  intermediate  tests,  experiments  and 
capability  demonstrations. 

•  The  second  paradigm  requires  the  same  M&S  teams  to  concurrently 
develop  a  robust  collection  of  simulation  environment  tools  for  SBA 
contract  delivery. 

•  A  proposed  tailoring  of  the  Federation  Development  and 
Execution  Process  (FEDEP)  is  set  forth,  capturing  the  maturation 
of  requirements  within  a  Spiral  Lifecycle  Model  (SLM),  allowing 
these  two  paradigms  to  co-exist  over  the  lifecycle  of  a 
development  program,  making  the  SBA  process  more  effective. 
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From  Science  to  Solutions 


Goals  and  Process 


VP 


•  Goals 

•  Identify  alignment  of  FEDEP  with  program  systems  engineering  processes 

•  Fill  gaps  to  create  robust  federation  engineering  process 

•  Share  lessons  learned  with  other  large  SBA  programs 

•  Process 

•  Bottom-up  mapping  of  FEDEP  artifacts  to  existing  program  artifacts 

•  Look  for  gaps  in  federation  artifacts  to  improve  FCS  process 

•  Top-down  mapping  of  planning  conferences  through  anchor  points  to  FEDEP 
steps/activities/tasks 

•  Identify  how  information  and  artifacts  must  flow  from  the  customer  through  the  systems 
engineering  processes  into  the  final  federation  artifacts 

•  While  anchor  points  are  not  used  by  many  programs,  the  concept  and  process  for  performing 
this  mapping  are  broadly  applicable 

•  Identify  opportunities  for  reuse  of  artifacts  from  IP  to  IP,  minimizing  rework 

•  Applicable  to  any  large,  iterative  SBA  program 

•  Make  recommendations  for  additions/modifications  to  program  processes  and 
artifacts 

•  While  some  of  these  lessons  learned  are  specific  to  FCS,  many  are  broadly  applicable 
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From  Science  to  Solutions 


Processes  to  Be  Aligned  $«««■» 


•Planning  conferences 

•System  of  systems  (SoS)  spiral  lifecycle  model 
(SLM)  anchor  points  (APs) 

•  Federation  Development  and  Execution 
Process  (FEDEP) 
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From  Science  to  Solutions 


Army  M&S  Specific  Guidance 
for  Planning  Conferences 


IPC 

MPC 

FPC 

•  Define  the  scenario:  Terrain, 
ORBAT  and  Campaign  Plan 

•  Define  command  and  control 
(Exercise  Control  Cell) 

•  Define  manning  for  exercise 
players,  response  cells,  control 
and  support 

•  Define  C4I  requirements 

•  Develop  the  training  plan 

•  Establish  database  milestones 
and  begin  build 

•  Determine  real-world  logistical 
support 

•  Draft  Memorandum  of 

Agreement  (MOA)  or  Pro  Forma 

•  Schedule  supporting  training 
events: 

•  Site  survey.  (pre-MPC) 

•  Database  builds  (including  ‘Good 
Idea  Cut-off  Time’) 

•  Scenario  Development  (pre-MPC) 
and  scripting  (post-MPC) 

•  ‘Train-the-trainer’  for  the  model 
and  ABCS  (post-MPC) 

•  Joint  and  outside  agency 
participation 

*  Present  coordinated  Exercise 
Plan  to  the  exercise  director 
and  senior  reps  from  key 
organizations: 

•  training  objectives 

•  exercise  objectives 

•  organizations  involved  and 
roles/responsibilities 

•  exercise  directive  (specified  tasks 
and  coordinating  instructions) 

•  planning  timeline,  tasks  required 
and  tracking  status 

•  scenario  progress,  ‘Road-to-War’, 
inject  requirements 

•  technical  plan,  database 
requirements,  simulation 
workarounds, 

•  budget  and  contract  requirements 

•  logistical  support 

•  cell  structure  and  manning 
requirements 

•  communications  plan 

•  O/C,  AAR  and  collection  plan 

•  Identify  cell  OICs 

*  Present  final  coordinated  plan 

•  Publish  FRAGO  if  required 

•  Review  MOA  milestones,  update 
status 

•  Resolve  outstanding  issues 

•  Review  training  objectives 

•  Review  manning 

•  Review  conduct  of  the  exercise 

•  Publish  Exercise  Control  Group 

•  Review  exercise  budget  versus 
changes  to  projected  costs 

•  Cell  OICs  present  and  provide 
backbriefs 

•  Review  training  requirements 
(O/C,  unit,  operator)  prior  to 
exercise 

•  OPFOR  review 
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to  © 


Spiral  Lifecycle  Model  (SLM)  & 


5.  REVIEW 
progress, 

CON  FI  RM 
commitment  to 
continue 


1.  ANALYSI S: 

Establish 

objectives, 

constraints, 

alternatives  for 

phase 


4.  PLANNj  NG: 
Plan  next  phases 


2.  DESIGN: 

Evaluate  product 
and  process 
alternatives, 
identify  and  resolve 
risks 


3.1  IMPLEMENTATION 
Develop,  verify  next- 
level  product 


(Boehm  1988) 
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From  Science  to  Solutions 


SLM  Tailoring  Process 


Driven  By: 

Succoss-Criticdl 
Stakeholder's 
Win 

Conditions 


Progress  Through  Steps 


2a.  Evaluate 
Alternatives  with 
Respect  to  Objectives, 
Constraints,  and  Priorities 


Risk 
Management 


Stakeholders 

Commitment 


Spiral 

Anchor-Point 

Milestones 


Feasibility 

Rationale 


LCA:  Life-Cycle  Architecture 
LCO:  Life-Cycle  Objectives 


At  a  SoS  level,  the 
SLM  might  be  tailored 
for  a  program  to 
include  the  following 
reviews  which  would 
occur  during  each 
phase  of  a  SLM  based 
program: 

•  Definition  Anchor  Point 

(DAP) 

•  Planning  Anchor  Point 

(PAP) 

•  Readiness  Anchor 
Point  (RAP) 

•  Assessment  Anchor 
Point  (AAP) 


(Boehm  2004) 
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From  Science  to  Solutions 


Anchor  Point  Definitions  &*»*»*» 


•  Definition  Anchor  Point  (DAP) 

•  Guidance  for  each  phase  focusing  more  and  more  on  end-state 
product 

•  Planning  Anchor  Point  (PAP) 

•  High-level  review  of  the  plans,  architecture,  and  risks  for  the  entire 
spiral  development 

•  Detailed  plans  of  the  specific  phase  in  question 

•  Risks  and  integration  challenges  identified  in  the  DAP 

•  Readiness  Anchor  Point  (RAP) 

•  Most  significant  checkpoint  associated  with  each  build 

•  Represents  commitment  across  all  levels  of  a  program  that  the  software 
build  can  be  successfully  implemented  within  the  build  budget  and 
schedule  using  the  documented  architectures  and  designs 

•  Risk  mitigation  plans  exist  for  all  potential  shortfalls 

•  Assessment  Anchor  Point  (AAP) 

•  Identify  process  improvements  that  can  be  made  in  subsequent 
phases 
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From  Science  to  Solutions 


Federation  Development  and 
Execution  Process  (FEDEP) 


Rather  than  dictating  a  "one-size-fits  all"  solution  for  all 
users,  the  FEDEP  provides  a  common  overarching 
process  framework  into  which  lower-level  domain- 
specific  management  and/or  engineering  methodologies 
can  be  easily  integrated. 


(IEEE  1516.3) 
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From  Science  to  Solutions 


FEDEP  Alignment  to  FCS 
Milestones 


Spiral  Lifecycle  Model 
SE/SD  Activities 


Tailored  Objectives  Concept 
FEDEP 


BE  Design 


BE  Develop 
«  Delta  Concept 


Event  Integrate 
BF  Design 


Execute 
BF  Develop 


Event  Integration  I  Execute  Analyze  & 

_ Report _ 


Phase 
Functionality 

-  UCs 

AOs 

& 

VOs 


o 

8-w 

o 


OCD 


Build  Early 
(BE) 


Build  Final 
(BF) 


Federate  BF  Development 


F 

— > 

External 

Labs 

E 

D 

$ 

1 

F 

N 

E 

T 

D 

1 

E 

G 

1 

-/ 

Major 

R 

Q 

Event 

A 

T 

T 

1 

1 

0 

N 
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From  Science  to  Solutions 


Top-down  Mapping  Results 


•  One  of  the  key  goals  of  this  paper  was  to  identify  how  the  FEDEP 
might  be  used  in  large  SBA  programs  such  as  FCS. 

•  Key  to  this  is  understanding  how  systems  engineering  processes 
interact. 

•  We  mapped  the  planning  conferences  through  the  anchor  points 
to  the  FEDEP  recommended  inputs,  tasks,  and  outcomes. 

•  Anchor  points  represent  gating  conditions  or  controls  on  the  FEDEP, 
not  technical  inputs. 

•  To  complete  this  mapping,  we  defined  four  new  anchor  point  credentials 
focused  on  reviewing  planning  conference  outputs  as  inputs  to  the 
FEDEP. 

•  Where  do  the  planning  conferences  and  anchor  points  impact  the 
FEDEP? 
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From  Science  to  Solutions 


FEDEP  Concurrence  Points 


•  1.1  -  Identify  User/Sponsor  Needs 

•  Any  known  constraints  which  may  affect  how 
the  federation  is  developed  and  executed 
(e.g.,  due  dates,  security  requirements) 

•1.2-  Develop  Objectives 

•  Assess  federation  feasibility  and  risk. 

•  Define  and  document  an  initial  federation 
development  and  execution  plan. 

•  Develop  initial  planning  documents,  including: 
Federation  development  and  execution  plan 
showing  an  approximate  schedule  and  major 
milestones. 

•  2.1  -  Develop  scenario(s) 

•  Federation  scenario(s) 

•  2.2  -  Develop  federation  conceptual  model 

•  Federation  conceptual  model 

•  2.3  -  Develop  federation  requirements 

•  Federation  requirements 

•  Federation  test  criteria 


•  3.2  -  Prepare  federation  design 

•  Federation  design 

•  Federation  architecture  (including  supporting 
infrastructure  design) 

•  Implied  requirements  for  federate 
modifications  and/or  development  of  new 
federates 

•  3.3  -  Prepare  plan 

•  Integration  plan 

•  4.1  -  Develop  FOM 

•  FOM 

•  FED/FDD 

•  5.3  -  Test  Federation 

•  Tested  (and  if  necessary,  accredited) 
federation 

•  7.2  -  Evaluate  and  feedback  results 

•  Lessons  learned 

•  Final  report 

•  Reusable  federation  products 


These  all  represent  points  where  the  federation  touches  the 
SoS,  but  where  M&S  specific  guidance  is  needed. 
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From  Science  to  Solutions 


Findings 


•  Recognize  the  federation  as  a  first-class  object. 

•  The  tested  federation,  the  output  of  FEDEP  activity  5.3,  must  be  a 
deliverable  itself . 

•  Program  level  guidance  needs  to  be  translated  into  executable,  M&S 
specific-guidance. 

•  Most  of  our  testing  plans  focus  on  testing  the  SoS  using  the  federation,  but 
there  is  very  little  information  on  testing  the  federation. 

•  Record  both  the  decisions  that  are  made  and  the  processes  by  which 
decisions  are  made. 

•  You  may  have  to  revisit  those  decisions  in  a  later  iteration,  e.g.  selection  of 
existing  federates  to  meet  the  requirements  of  a  particular  iteration. 
Knowing  the  criteria  for  the  decision  can  expedite  reevaluation. 

•  Federation  requirements  must  be  readily  identifiable  as  a  subset  of 
SoS  requirements. 

•  Additionally,  there  should  be  continuous  requirements  management 
because  delays  in  delivery  of  operational  software  may  require  filling  in 
those  items  with  M&S,  but  that  too  takes  time. 
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From  Science  to  Solutions 


Findings 


•  Recognize  where  M&S  is  the  same  and  where  it’s  different  from  your 
operational  software. 

•  For  example,  non-operational  M&S  may  not  need  as  rigorous  testing  as 
operational  software,  but  the  same  CM  and  documentation  standards 
probably  apply. 

•  However,  consider  the  global  implications  of  relaxing  standards  for  M&S 
because  it  may  have  broader  implications,  e.g.  reducing  the  level  of  testing  for 
M&S  may  reduce  your  ability  to  fully  test  operational  software  that  depends  on 
M&S. 

•  Embedded  M&S  is  operational  software  and  should  be  treated  as  such. 

•  M&S  can  solve  your  representation  shortfalls,  not  your  interface  ones. 

•  M&S  has  to  have  interfaces  too,  preferably  the  same  ones  used  for 
operational  systems  so  operational  code  can  be  dropped  in  readily  later. 
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From  Science  to  Solutions 


Estimating  Rework  Across  IPs 


•  Reviewed  each  FEDEP  recommended  task  and  estimated  level  of  rework  in  later  IPs  based  on  the 
assumption  that  preceding  IPs  were  successfully  executed  and  assigned  the  following  values: 

•  1  (green)  -  little  to  no  rework  in  subsequent  iterations.  Either  program  level  documents  remain  essentially  unchanged  or 
a  program  process  is  already  in  place  that  minimizes  effort. 

•  2  (yellow)  -  some  rework,  but  not  a  substantial  engineering  effort.  Additional  or  updated  entity  or  scenario 
representations  necessitate  engineering  effort  that  ripples  throughout  federate  and  federation  engineering. 

•  3  (red)  -  significant  rework.  The  actual  federate  and  federation  engineering  required  to  implement  new  functionality  that 
represents  the  core  of  the  iteration  intent. 

•  Rolled  up  statistics  to  FEDEP  tasks  and  further  into  FEDEP  steps 


Irer  .irian 
Rework, 


FEDEP  iteps 

Description 

Mecric 

1 

Define  Federation  Objectives 

Define  and.  dacument  a  set  of  needs  that  are  to  be  addressed  through  the 
develop inenr  and  execnri&n  of  an  HLA  federation  and  to  transform  these  needs  into 
a  mote  detailed  list  of  specific  fed  era  den  objectives 

1 

1.1 

Identity  U ser  Sponsor  2i" eeds 

Develop  a  clear  under:  tar.  dins  of  the  picblem  to  be  addressed  by  the  federation 

1.1.2 

1.1.2. 1 

Recommended  tasks 

Identify  program  objectives  that  motivate  federation  development 

1 

1 

1.1. 2.2 

Identify  available  resources  and  know::  development  and  exec  tit  loss  constraints 

2 

1 .1.2. 3 

Document  this  information  in :.  needs  statement 

i 

1.2 

Develop  Objectives 

1.2.2 

Recommended  Tasks 

1.2.2. 1 

Analyse  the  needs  statement. 

1 

1 ,2_2.2 

Assess  federation  feasibility  and  risk. 

2 

1.2. 2. 3 

122  A 

Define  and  document  a  prioritized  set  o:  federation  obj  ectLves.  consistent  with  the  needs 
statement. 

Meet  with  tnE  federation  sponsor  to  review  the  federation  objectives,  and  reconcile  any 
differences. 

1 

I 

122.5 

Define  and  document  an  initial  federation  development  and  Execution  plan. 

7 

122.6 

Identify  potential  tool  s  to  support  die  initial  plan. 
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High  Level  Rework  Analysis 

Results* 


*Detail  by  FEDEP  step  in  backup  slides; 
detail  by  task  in  07F-SIW-083 


Approved  for  Pwiblc  Release,,  Elstefouillfoini  Uimliimtedl,  PM  PCS  22  Oct  2007,  case  07-271 . 


16 


5S!" 


From  Science  to  Solutions 


Final  Thoughts 


•  Program-level  processes  focus  on  cost,  schedule,  and 
risk. 

•  From  the  SoS  perspective,  the  federation  (and  the 
FEDEP)  are  test  tools. 

•  The  FEDEP  is  focused  on  the  technical  aspects  of 
producing  a  federation. 

•  For  FCS,  we’re  introducing  criteria  for  gating  program- 
level  processes  down  to  the  FEDEP  to  align  these 
different  focuses. 

•  For  the  broader  SBA  community,  we’re  providing  input 
to  the  SISO  update  of  the  FEDEP  to  introduce  the 
hooks  necessary  for  a  large,  iterative,  SBA  program. 
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Backups 


Detailed  Artifact  Rework  Analysis 
by  FEDEP  Step 
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Step  1  -  Define  Federation 

Objectives 
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Step  2  -  Perform  Conceptual 

Analysis 
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Step  3  -  Design  Federation 


Federation 


Approved  for  Pwiblc  Release,,  Elstefouillfoini  Uimliimtedl,  PM  PCS  22  Oct  2007,  case  07-271 . 


22 


From  Science  to  Solutions 


Step  4  -  Develop  Federation 
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Step  5  -  Plan,  Integrate  and 
Test  Federation 


Federation  Agreements 

r 

\ 

Scenario  Instance(s) 

Plan 

Execution 

FDD 

Federation  Development  Plan 

5.1 

FOM 

Federation  Environment 
Description 


Federation  Environment 
Description 


r 

\ 

List  of  Selected  (Existing)  Federates 

Integrate 

Federation 

Modified  /  New  Federates 

Implemented  Federation  Infrastructure^ 

5.2 

Supporting  Databases 

- ► 

_ / 


Integrated 

Federation 


r 


Federation  Test  Criteria 


Federation  Development  Plan 


Federation  Agreements 


Test 

Federation 

5.3 
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Tested  Federation 

- ► 
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Step  6  -  Execute  Federation 
and  Prepare  Outputs 


Documented  Execution 


Derived 

Outputs 


*■ 
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Step  7  -  Analyze  Data  and 
Evaluate  Results 


Lessons  Learned 

- ► 

Final  Report 

- ► 

Reusable  Federation  Products 

- ► 
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“I  have  made  this  letter 
longer  than  usual 
because  I  lack  the  time 

to  make  it  shorter” 

Blaise  Pascal 
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World-Class  Quality 


Provide  motivation  and  principles  for  lean, 
maintenance,  and  service. 

Describe  Service/Maintenance  in  terms  of 

“projects”  and  CMMI®. 

Describe  successful  CMMI  Tailoring  for 
Service/Maintenance  Organizations. 

Answer  any  of  your  questions. 


CMMI  is  a  registered  trademark  in  the  US  Copyright  and  Patent  Office  by  Carnegie  Melon  University. 
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Agenda 


Introduction 


Motivation  and  Background 
Tailoring  Project  Management 
Tailoring  CM 
Tailoring  Engineering 
Miscellaneous  Tailoring 
Questions  and  Answers 
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Lean  Problems 


World-Class  Quality 


Most  organizations  have  too  much  waste  (e.g., 
non-value  added). 

Most  processes  have  too  many  “non-value 
added”  steps. 

How  can  organizations  focus  on  “value  added” 

and  remove  waste? 

How  can  organizations  measure  value  and 
waste? 

Lean  is  a  recent  quality  approach  to  help 

organizations  focus  on  “value”  and  remove 
“non-value”. 
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What  is  Lean? 


World-Class  Quality 


Lean  has  its  roots  in  quality  and  manufacturing, 
and  is  a  recent  popular  movement  in  quality. 

“Lean  Production”  is  the  name  for  the  Toyota 

Lean  Production  System. 

The  following  are  major  lean  references  (books): 

-“The  Machine  That  Changed  The  World” 
-“Learning  to  See” 

-“The  Toyota  Way” 

-“The  Toyota  Product  Development  System” 
-“Lean  Thinking” 
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World-Class  Quality 


Establish  customer  defined  value  (i.e.,  identify 

the  “value  stream”).  Process  =  “value”. 

Continuously  eliminate  non-value  added 
activities  (e.g.,  waste,  rework,  defects). 

Use  leadership  and  standardization  to  create  a 
lean  culture. 

Align  your  organization  through  visual 
communication. 

Create  an  optimized  process  flow  (e.g.,  “Flow”, 
“Pull”,  “Just-In-Time”,  “Leveled”). 
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World-Class  Quality 

^^Some  Lean  Principles  - 

Use  lean  metrics  to  m£ra^je  the  value  stream. 

Front-Load  the  process  for  maximum  design 
space. 

Build  a  learning  organization  to  achieve  lean 
and  continuous  improvement. 

Adapt  technology  to  fit  your  people  and 
processes. 

Strive  for  perfection  through  continuous 
improvement. 
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Agenda 


Introduction 


Motivation  and  Background 
Tailoring  Project  Management 
Tailoring  CM 
Tailoring  Engineering 
Miscellaneous  Tailoring 
Questions  and  Answers 
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ome 


Service/Maint .  Successes 


HP  Success  Story 

-  Lean  CMMI®  L3  Process  25%  of  the  size  of  HP  India  Process 

-  Very  Small  Projects  (0.25-0.5  FTE  projects) 

-  Includes  website  development 

-  Includes  maintenance/service 

-  See  References  [Kellum  2006] 


Raytheon/NASA  JPL  Success  Story 

-  Documented  in  SEI  Report 

-  Tailored  all  CMMI  L3  practices  in  report 

-  Only  one  customer  (JPL)  -  Simple  model 

-  See  References 


Numerous  CMM  Success  Stories 

-  More  and  more  CMMI  service/maintenance  success  stories  emerging 

Draft  CMMI®  for  Service 

-  Has  not  been  released  by  SEI 
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Agenda 


Introduction 

Motivation  and  Background 


Tailoring  Project  Management 
Tailoring  CM 
Tailoring  Engineering 
Miscellaneous  Tailoring 
Questions  and  Answers 
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T53 


Project  Management  PAs 


Project  Planning  (PP  -  L2) 

Project  Monitoring  and  Control  (PMC  -  L2) 

Integrated  Project  Management  (IPM  -  L3) 
-Tailoring 

-Advanced  Project  Management 
Risk  Management  (RSKM  -  L3) 

Supplier  Agreement  Management  (SAM  -  L2) 


•  Reference:  ^CMMIsm  for  Systems  Engineering  and  Software  Engineering,^  CMMI-SE/SW  Staged  Version,Version  1.1 
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World-Class  Quality 


T53 


Project  Planning  Goals 


SG  1 :  Estimates  of  project  planning  parameters 
are  established  and  maintained. 

SG  2:  A  project  plan  is  established  and 
maintained  as  the  basis  for  managing  the 
project. 

SG  3:  Commitments  to  the  project  plan  are 
established  and  maintained. 


•  Reference:  “CMMIsm  for  Systems  Engineering  and  Software  Engineering,”,  CMMI-SE/SW  Staged  Version,  Version  1.1 
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Commitment  Metrics 


Plate 

Full 

COMMITS 

Size 

Effort 

Cost 

Schedule 

Defects 

1. 

2. 

3. 

■ 

■ 

■ 

N 

Backlog 

N+1 

■  ■  ■ 
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World-Class  Quality 

^^ervice/Maint .  Principles 

The  CMMIis  “project”  oriented.  In  a  Service/ 

Maintenance  organization,  there  may  not  even  be 

a  “project”. 

The  term  “Project”  may  not  work  in  your 
organization  (“Project”  definition:  Start  Date/End 

Date).  This  can  be  a  major  problem  when 
interpreting  the  CMMI  for  Service/Maintenance. 

Most  of  the  time,  there  are  a  collection  of 
activities  that  can  be  grouped  together: 

-  Releases/Bundles 
-Tasks/Service  Requests 

-  Change  Requests/Problem  Reports 

-  Annual  Plans  (e.g.,  service,  maintenance) 
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e  Service/Maint .  Plans 


World-Class  Quality 


Possible  Equivalents  to  CMMI  Project  Plans: 

•  Release  Plan  (e.g.,  Annual,  Quarterly,  Monthly) 

•  Task  Plan  (e.g.,  for  a  customer  under  a  PO) 

•  Service  Request 

•  Service  Level  Agreement  (SLA) 

•  Annual  CM  Plan 

-  Change  Requests(CRs)/Problem  Reports  (PRs) 

•  Annual  Plans  (e.g.,  service,  maintenance) 

-  Can  have  releases,  CRs/PRs,  Service  Requests,  Tasks 
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World-Class  Quality 


Service /Main t . 


PM 


Tailoring 

Don’t  change  your  business  to  match  CMML 

Improve  your  business  to  meet  CMMI  Goals. 


Use  Lean  Templates  that  implement  CMMI 
requirements  (combine  items). 


Put  many  ofthe  CMMI  requirements  that  don’t 

change  across  tasks  in  annual  plans  (e.g.,  Scope, 
Data,  Training,  Risks,  CM,  etc.)  The  things  that  do 
change  (e.g.,  estimates,  schedule,  etc.),  make  lean 
and  tailor  to  tasks. 


Schedules  can  be  very  different  (e.g.,  more 
focused  on  releases/CM  than  milestones). 
Tracking  can  be  done  periodically  (e.g.,  monthly), 
and  meetings  may  be  combined. 
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Agenda 


Introduction 

Motivation  and  Background 
Tailoring  Project  Management 


Tailoring  CM 
Tailoring  Engineering 
Miscellaneous  Tailoring 
Questions  and  Answers 


Training  Material  Used  with  Permission  and  Licensed  by  Lean  Solutions  Institute,  Inc.  (LSI) 


18 


The  Customer  and  CM 


World-Class  Quality 


Why  perform  CM? 

If  effective  CM  is  not  performed,  the  risk  of  shipping 
the  wrong  version  to  a  customer  is  too  high.  For 

example,  a  version  delivered  to  a  customer  might ... 

-  Have  defects 

-  Have  untested  changes 

-  Not  be  reproducible 

CMis  all  about  “Product  Integrity”: 

-  Knowing  exactly  what  customers  have 

-  Knowing  the  exact  status  of  products,  versions, 
baselines,  configuration  items  (e.g.,  exactly  what  is  in 
which  version) 

-  Knowing  how  to  reproduce  every  product,  version, 
component,  configuration  item,  etc. 
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World-Class  Quality 


CM 


Configuration  Identification: 

•  What  are  configuration  items,  and  what  configuration 
does  your  customer  have? 

Configuration  Control: 

•  How  do  I  control  changes  made  to  the  configuration? 

Configuration  Status  Accounting: 

•  What  is  the  current  status  of  the  configuration? 

Configuration  Auditing: 

•  Does  the  configuration  have  product  integrity? 


•  Reference:  “Configuration  Management”  Training;  Copyright  ©  by  Process  Assets,  LLC  (PAL). 
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CMMI  CM 


World-Class  Quality 


SG  1 :  Baselines  of  identified  work  products  are 
established 


SG  2:  Changes  to  work  products  under 

configuration  management  are  tracked  and 
controlled 


SG  3:  Integrity  of  baselines  is  established  and 
maintained 


•  Reference:  CMMhu for  Systems  Engineering  and  Software  Engineering,  CMMI-SE/SW  Staged  Version,  Version  1 . 1 
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NASA  JPL  MGSS  CM  Process 


7.1  Perform 

CM  Planning 

7.2  Perform 

Configuration 

Control 

7.3  Perform 
Configuration 

Status 

Accounting 

7.4  Perform 

CM 

Audits 

Training  Material  Used  with  Permission  and  Licensed  by  Lean  Solutions  Institute,  Inc.  (LSI) 


22 


World-Class  Quality 


Example  CM  Process 


5  W’s  on 
1  Page 
in  a 

Process 

Model! 


Patent 

Pending 

Approach 


7.1  Perform  CM  P  Ian  n  i  ng 

PURPOSE  :  Develop  CM  Plans  th  at  m  e  et  project  needs. 

INPUTS/EN  T  RY  CR  ITERIA 

CONTEXT 

•  Project  initiated  AND  Draft  Project  PI  an 

•  Organiz  a  ti  o  nal  CM  PI  a  n  is  a  pproved  and  m  e  ets  CM  Standard. 


ROLES  &ACTIV  I  TIES: 

Pr  oj  ect 
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World-Class  Quality 

^  Service  CM  Tailoring 

What  is  the  operational  definition  of  a 

“project”? 

Ho  w  big  does  a  “project”  need  to  be  before  it 

can  handle  the  overhead  of  the  CMMI? 


Small  Change  Requests  (CRs)  and  Problem 
Reports  (PRs)  are  what  CM  is  all  about. 

How  do  you  plan  for  interrupt  driven  CRs  and 
PRs?  (e.g.,  you  know  the  customer  will  make 
changes  and  you  know  there  will  be  defects) 

Budget  for  CRs/PRs  based  on  historical  data  in 
an  Annual  Plan. 
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Annual  CM  plan 


World-Class  Quality 


$0.00 

Actual 

Budget 

Here 

Total  CRs/PRs  completed  to  date 
(Total  hours  or  total  budget) 

Planned 

Budget 

Here 

Planned  CRs/PRs  (Not  completed) 

Total  $$$ 
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Engineering  Process  Areas 


Customer  needs 


\ 

\ 

i 


i 


/ 


•  Reference:  “CMMIsm  for  Systems  Engineering  and  Software  Engineering,”,  CMMI-SE/SW  Staged  Version,  Version  1.1 


Training  Material  Used  with  Permission  and  Licensed  by  Lean  Solutions  Institute,  Inc.  (LSI) 


27 


World-Class  Quality 

Tailoring  Engineering 

For  large  or  medium  projects  (e.g.,  large 
tasks/service  requests/change  requests),  the 
CMMI  can  be  used  effectively. 


For  small  and  very  small  stand-alone  tasks,  the 
CMMI  engineering  process  areas  have  a  lot  of 

overhead.  One  solution  is  a  “mini-spec”. 


For  very  small  tasks,  sometimes  it  is  better  to 
run  them  under  CM  as  a  CR/PR  and  not  formally 

define  them  as  a  “project”. 

If  desired,  CM  systems  can  be  made  to  handle 
CMMI  requirements. 
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Example  CR/PR  States 
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Example  Requirements 

Matrix 


# 

Requirement 

Reference 

Allocation 

Stability 

(H/M/L) 

Risk 

(H/M/L) 

Priority 

(H/M/L) 
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^  Other  CMMI Processes 

Project  Management,  Engineering,  CM  - 
Covered. 


Process  Management  PAs  (i.e.,  OPD,  OPF,  OT, 
OPP,  OID)  apply  well  to  Service/Maintenance 
organizations  because  they  are  at  the 

organizational  level  (not  the  “project”  level). 

Most  support  process  areas  (i.e.,  PPQA,  DAR, 
CAR)  also  apply  well  to  Service/Maintenance 
organizations  because  they  are  like 
organizational  level  processes  (e.g.,  supporting 
projects). 

Metrics  (e.g.,  MA,  QPP)  at  the  project  level  need 
to  be  tailored  to  Service/Maintenance 
organizations. 
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Lean  Measurement  FrameworkSM 


Based  on  three  industry  best  practices  (will  be 
presented  on  next  few  slides). 

Helps  organizations  focus  on  the  “vital  few” 

metrics. 

Based  on  the  three  primary  usage  scenarios  for 
metrics. 

Based  on  metrics  that  are  strongly  supportive 
of  goals  and  objectives. 

Award  winning  measurement  framework  from 
American  Society  for  Quality  (ASQ). 
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Lean  Measurement  FrameworkSM 


GOALS 

KEY  QUESTIONS 

METRICS 

DC 

DS 

PLAN 

Cost,  defects, 
effort,  size, 
schedule,  etc. 

CONTROL 

Cost,  defects, 
effort,  size, 
schedule,  etc. 

IMPROVE 

Cost,  defects, 
effort,  size, 
schedule,  etc. 

•  DC  =  Data  Collection;  DS  =  Data  Storage 
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Some  Example  Lean  Metrics 


Takt  Time 
Lead  Time 
Process  Time 
Changeover  Time 
Available  Time 
Value-Added  Time 
Demand  Rate 
Number  of  People 
Inventory 
Percent  Complete  and  Accurate 
Information  Technology  Used 
Reliability 


Data  Box 
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"Top  Down" 


Organizational  vision,  mission,  and  strategy 
should  drive  metrics.  Metrics  should  be  driven 
by  and  connect  to  goals  and  objectives. 

Strategic  Planning  should  identify  measurable 

success  criteria  and  measurable  objectives: 

•  Compliance  (e.g.,  Government  requirements) 

•  Industry  Standards  (e.g.,  Baldrige,  CMMISM,  ISO,  etc.) 

•  Market  Share 

•  Performance  (e.g.,  CPI,  SPI) 

•  Productivity 

•  Quality 

•  ROI 

•  Time  to  market 
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Project  management  processes  need  the  most 
tailoring. 

CM  is  a  strong  service/maintenance  process  - 
use  it! 

Engineering  processes  need  to  be  tailored  to 
service/maintenance  (e.g.,  small  projects). 

Organizational  and  support  processes  work  well 
for  service/maintenance. 
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Define  an  operational  definition  of  a  “project”. 

CMMI  and  processes  must  be  tailored  to 
service/maintenance  organizations. 

Implement  a  lean  solution  (e.g.,  lean  processes, 
procedures,  templates,  etc).  Many  CMMI 
implementations  are  NOT  lean. 

Not  every  part  of  business  needs  to  be  under 
CMMI  (only  do  what  makes  business  sense). 

Make  a  “project”  large  enough  to  handle  CMMI 
overhead  (i.e.,  should  make  business  sense). 
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Radars:  SPS-49,  SPS-48, 
SPQ-9B ,  MFR... 


CIWS/SEARAM  sensor 
ES,  IRST 

SLQ-32,  advanced  ES 


DEW 


CEC,  OATM 


Signature  control 


Operational  Context: 
Ship  Self  Defense 


Ship  Defense  MOE 

Probability  of  Raid  Annihilation  (PRA) 

is  the  ability  of  a  particular  stand-alone  ship  as  a  system  to  detect, 
control,  engage,  and  defeat  a  specified  raid  of  threats  within  a 
specified  level  of  probability  in  an  operational  environment 


NATO  Seasparrow, 
ESSM 


Onboard  EA 


MK  214 
Chaff 


Multi-threat  raid 


MK  216 
Chaff 


Subsonic,  supersonic, 
high  diver 

Hi-G  maneuvers 
Multi-mode  seekers 


TimeUnt a  30  seconds 


Battle 

Space 


0-12  nmi 


Enterprise  Test  &  Evaluation 

Master  Plan 


UNCLASSIFIED 


(V)  Test  ami  Evaluation  Master  Plan 
No.  1714 

Capstone  Enterprise  Air  Warfare 
Shij)  Self-Defense 


The  purpose  of  the  Capstone  Enterprise  Air 
Warfare  Ship  Self  Defense  (AW  SSD)  Enterprise 
Test  and  Evaluation  Master  Plan  (TEMP)  is  to 
consolidate  all  AW  SSD  at-sea  testing  and  PRA 
Testbed  testing 


2?  May  2006 


Deparfmenr  of  fhe  Navy 

peo  nre 


The  AW  SSD  T&E  Enterprise  Strategy  is  founded 
on  a  two-tiered  process  to  assess  AW  SSD 
warfare  systems  performance: 

1)  Validate  models  with  live  testing 

•  Operational  Ship  testing 

•  Self  Defense  Test  Ship  (SDTS)  testing 

2)  Assess  performance  with  models 


Test  Events  DTIOT-ET15  thru  ET19 
are  formal  PRA  Testbed  events 

Includes  DDG  1000,  LHA  6,  LCS  and 
CVN  21  ship  classes 
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Effective  date:  24  October  2007 


Enterprise  Pra  Testbed  System  Engineering  - 
Drivers  for  Centralized  IWS  Leadership 


•  Systems  performance  for  PRA  assessment  spans 
different  technical  communities  and  multiple  managing 
program  offices 

*  PRA  will  be  assessed  using  a  federation  of 
interoperable  simulations;  it  will  not  (cannot)  be  tested 
empirically 

-  Complex,  multi-spectral,  integrated  HK/EW  problem  space 

•  Many  specific  parameters,  assumptions,  and 
limitations  are  negotiated  between  the  testing  and 
acquisition  communities 

*  The  testing  community  is  intent  on  consistent  PRA 
assessment  across  ship  classes  and  warfare  system 
configurations 

-  Different  hulls,  different  configurations. .  .same  threat  models,  same 
virtual  range  conditions 
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Enterprise  Test  Planning  &  Execution 


•  Non-traditional  factors 

-  M&S  events  as  formal  test  events 

•  “Virtual  Range”  requirement 

-  Expectation  for  formal,  planned  data  flow  from  empirical 
testing  to  model  validation 

•  Organization  and  planning  are  combat-system¬ 
centric  vice  platform-centric 

-  Single  Enterprise  Test  Team 

-  Centralized  management  and  resourcing  of  PRA  Testbed 

-  Multiple  ship  classes  provide  testing  data  supporting 
PRA  Testbed  component  development  and  validation 
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EXtCUl 


Navy  Ship  Self  Defense 
T&E  Enterprise  IPT  Structure 


SSD  T&E  Enterprise  IPT 

Chair:  PEO  IWS 
Representatives: 

•  DOT&E  •  IWS  WSEs  •  N7 

•  COTF  •  Ship  Class  Reps  •  N43 

•  OSD  (A  T&L)  •  IWS  MPM  Reps  •  N091 

•  SEA  06 
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Chair  N091 

Sub-group  chairs:  N43  for  targets ,  IWS  7D  for  models 
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EXtCtl] 


Enterprise  P^A  Testbed 
System  Engineering 


•  Engineering  one  Enterprise  Testbed,  which  is 
instantiated  in  several  unique  configuration  baselines 

-  Formally  accredited  Baselines  are  correlated  to  Enterprise  test  events 
and  ship  class  OPEVALs 

-  Element  Project  Offices  are  vendors  to  Enterprise  not  individual  ship 
classes 

•  One  master  set  of  requirements  for  the  Testbed 

-  Fed  by  both  Enterprise  SE  and  Baseline  IPTs 

-  Allocated  and  adjudicated  according  to  Enterprise  deliveries 

•  A  single  Enterprise  delivery  may  provide  capability  to 
more  than  one  Testbed  Baseline 

-  A  single  set  of  SE  artifacts  is  maintained  at  the  Enterprise  level 

•  Testbed-based  Enterprise  test  events  will  be  treated  as 
empirical  events 

-  E.g.,  test  readiness  reviews,  test  objectives 
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Enterprise  f*RA  Testbed 

Components 


“Virtual  Range” 


Testbed  Component 
Providers: 

|  IWS7D 


Ship  Class  PM 


IWS  Project  Offices 


“Virtual  Range”  (Infrastructure) 

•  Testbed  Architecture:  network 
interface  layer,  interface  standards, 
functional  allocation  standards 

•  Common  Threat  Models:  seeker, 
airframe/autopilot,  signatures, 
vulnerability 


L 

Navy  PRA  Testbed  Ship  Class  Baseline 


f 


Common  Environment  Models: 
tailored  authoritative  databases, 
runtime  environment  data  services 

Virtual  Test  Ship”(specific  to  ship 
class) 

Ship  Characteristics 


Ship-specific 

Characteristics 


1 

Combat 

Combat 

■  Combat 

1 

■  Combat 

System 

System 

i  System 

3  System 

Element 

Element 

1  Element 

1  Element 

“Virtual  Test  Ship” 


•  Signature,  motion,  launcher 
placements,  etc. 

Combat  System  Representation 

•  Authoritative,  “T&E  quality” 
models  of  combat  system 
elements 
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EXtCtl] 


Current  Simulation  Framework 

Characteristics 


•  HLA  federation  implementation 

-  All  system  representations  execute  simultaneously  for  each 
ship  defense  engagement 

•  Geographically  distributed 

*  Constructive  simulation,  conservative  time 
management 

*  System  representations  are  a  mix  of  digital  models 
and  tactical  software 

-  Most  representations  are  a  hybrid  of  tactical  SWIL  and  digital 
model 

-  Most  tactical  SW  re-hosted  to  general  purpose  computers 
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Pra  Testbed  Deployment 

LPD  17  Baseline 
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Enterprise  Pra  Testbed  Status 


•  PRA  Testbed  Configuration  Working  Group 
Established  Under  Ship  Self  Defense  T&E 
Enterprise 

-  Chaired  by  IWS  7D,  Shala  Malone 

-  Testbed  baseline  IPTs  established  for  current  Enterprise  ship 
classes:  LHA  6,  DDG  1000,  CVN  21,  and  LCS 

•  LPD  17  Testbed  Development  Underway  in 
Support  of  Ship  Class  OT&E 

-  4th  spiral  integration  underway 

-  LPD  17  assessment  ‘runs  for  score’  commence  1QFY08; 
completion  planned  for  CY08 
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Enterprise  Pra  Testbed  Evolution 


Consistent  Testbed  development 
across  ship  classes  and  CS  configurations 


Enterprise 
Pra  Testbed 
Baselines 

Common  architecture 
common  threats  & 
environment,  model  re-use 


PEO  IWS  7D 
Leadership 


LPD  17 

Testbed 

Baseline 


■  Validated  models, 
lessons  learned, 
arch,  advances 


“Sir 


LHA  6 
Testbed 
Baseline 


CVN  78 
Testbed 
Baseline 


Common  Virtual  Range 


Process  Standards 
&  Architecture 


Testbed  Configuration 
Management 


PEO  IWS 

Project  Offices 


SPS- 

48E 


SPS- 

49A 


ES 


CEC 


RAM 


CiWS 


SM-6 


SIAP 


SPQ- 

9B 


DBR 


Decoys 


SSDS 


TSCE 


ESSM 


Open 
Arch . 


Element  System  Representations 
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Significant  cost  avoidance  through 
re-use  of  models ,  virtual  range ,  &  architecture 


Effective  date:  24  October  2007 


Challenges  Ahead 


•  Feedback  of  knowledge  and  capabilities  to  early 
phase  acquisition  systems  engineering 

•  Improved  mechanisms  for  injecting  data  needs  into 
planning  of  empirical  tests 

•  Relationship  of  PRA  Testbed  simulations  to  other  M&S 
supporting  system  development  and  T&E 

•  M&S  capabilities  development  to  support  Family-of- 
Systems  development 
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Questions? 
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Objectives 


Describe  some  requirements  problems  from 
industry. 


Present  a  useful  classification  of  requirements 
problems. 


Describe  some  practical  strategies  and  best 
practices  that  organizations  have  used  to 
successfully  develop,  manage,  and  improve  their 
requirements  in  a  measurable  way. 

Provide  real  examples  that  address  requirements 
problems. 


Answer  any  of  your  questions. 
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Outline 


Why  Focus  on  Requirements? 


A  Practical  Requirements  Classification 
CMMI®  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by 

Training  Material  Used  with  Permission  and  Licensed  by  Lean  Solutions  Institute,  Inc.  (LSI) 


ie  Mellon  University 
Slide  3 


World-Class  Quality 


Why  Focus  on  Requirements? 


The  hardest  single  part  of  building  a 
system  is  deciding  what  to  build... 

No  other  part  of  the  work  so  cripples  the 
resulting  system  if  done  wrong.  No  other 
part  is  more  difficult  to  rectify  later. 


Adapted  from  Fredrick  Brooks,  Jar.  [Brooks  87] 
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Why  Focus  on  Requirements? 


A  research  report  from  the  Standish  Group 
highlighted  the  continuing  quality  and  delivery 
problems  in  our  industry  and  identified  three 
leading  causes: 

•  Lack  of  user  input 

•  Incomplete  requirements  and  specifications 

•  Changing  requirement  specifications 


•  Reference:  “Chaos”,  Compass,  The  Standish  Group,  1997,  used  with  permission. 
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Outline 

Why  Focus  on  Requirements? 


A  Practical  Requirements  Classification 
CMMI  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 
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Problems  with  Requirements 


According  to  the  SEI  [Christel  92],  problems  of 
requirements  elicitation  can  be  grouped  into  3 
categories: 


1 .  Problems  of  Scope:  the  requirements  may 
address  too  little  or  too  much  information. 


2.  Problems  of  Understanding:  problems  within 
groups  as  well  as  between  groups  such  as 
users  and  developers. 

3.  Problems  of  Volatility:  the  changing  nature 
of  requirements. 
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Scope  and  Volatility 


The  list  of  10  requirements  elicitation  problems 
given  in  [McDermid  89]  can  be  classified  according 
to  the  3  categories  in  [Christel  92]: 

Problems  of  Scope 

•  The  boundary  of  the  system  is  ill-defined 

•  Unnecessary  design  information  may  be  given 


Problems  of  Volatility 
•  Requirements  evolve  over  time 
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Problems  of  Understanding 


•  Users  have  incomplete  understanding  of  their 
needs 


*  Users  have  poor  understanding  of  computer 
capabilities  and  limitations 

•  Analysts  have  poor  knowledge  of  problem 
domain 


•  User  and  analyst  speak  different  languages 

•  Ease  of  omitting  “obvious”  information 

*  Conflicting  views  of  different  users 

*  Requirements  are  often  vague  and  untestable, 

e.g.,  “user  friendly”  and  “robust” 
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Outline 


Why  Focus  on  Requirements? 

A  Practical  Requirements  Classification 
CMMI  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 
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Requirements  Management  (REQM) 


SG  1 :  Manage  Requirements: 


SP  1.1-1:  Obtain  an  Understanding  of  the 

Requirements 

SP  1.2-2:  Obtain  Commitment  to  Requirements 

SP  1.3-1:  Manage  Requirements  Changes 

SP  1.4-2:  Maintain  Bidirectional  Traceability  of 

Requirements 


SP  1.5-1:  Identify  Inconsistencies  between 

Project  Work  and  Requirements 


•  Reference:  “Capability  Maturity  ModefMntegratjpn  (CMMI),  Version  1.1’\  CMU/SEI-2002-TR-01 1 ,  March  2002 
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Requirements  Development  (RD) 


SG  1 :  Develop  Customer  Requirements: 

SP  1.1-1:  Collect  Stakeholder  Needs 
SP  1.1-2:  Elicit  Needs 

SP  1.2-1:  Develop  the  Customer  Requirements 


SG  2:  Develop  Product  Requirements: 

SP  2.1-1:  Establish  Product  and  Product-Component  Requirements 
SP  2.2-1:  Allocate  Product-Component  Requirements 
SP  2.3-1:  Identify  Interface  Requirements 


SG  3:  Analyze  and  Verify  Requirements: 

SP  3.1-1:  Establish  Operational  Concepts  and  Scenarios 
SP  3.2-1:  Establish  a  Definition  of  Required  Functionality 
SP  3.3-1:  Analyze  Requirements 
SP  3.4-3:  Analyze  Requirements  to  Achieve  Balance 
SP  3.5-1:  Validate  Requirements 

SP  3.5-2:  Validate  Requirements  with  Comprehensive  Methods 

•  Reference:  “Capability  Maturity  ModeFjntegration  (CMI\%  Version  IT,  CMU/SEI-2002-TR-01 1 ,  March  2002 
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Engineering  Process  Areas 


Customer  needs 


•  Reference:  “Capability  Maturity  ModeJ®lntegration  (CMMI),  Version  CMU/SEI-2002-TR-01 1 ,  March  2002 
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CMMI  and  Requirements 


Requirement  processes  need  to  be  defined, 
trained,  and  improved  (e.g.,  OPF,  OPD,  OT,  OID). 


Support  processes  are  critical  for  measuring  and 
managing  requirements  (e.g.,  CM,  MA,  PPQA). 


Defects  need  to  be  removed  and  prevented  in 
requirements  (e.g.,  PI,  VER,  VAL,  CAR). 


IPPD  (i.e.,  integrated  product  teams)  also  contains 
allocating  requirements  to  teams  (e.g.,  IPM). 

Supplier  Sourcing  requires  managing  supplier 
requirements  (e.g.,  SAM). 
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Outline 


Why  Focus  on  Requirements? 

A  Practical  Requirements  Classification 
CMMI  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 
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Practical  Strategies 


1.  Define  a  lean  Requirements  Management  (REQM) 
Process. 


2.  Use  lean  Configuration  Management  (CM)  and  CM 
Metrics. 


3.  Use  Requirements  Metrics  (e.g.,  priority,  stability, 
risk,  number  of  requirements,  defect  density,  etc). 

4.  Define  the  requirements  process  (RD),  and  use 
lessons  learned  from  quality  (e.g.,  QFD,  Juran, 
etc). 

5.  Tailor  a  requirements  standard  (e.g.,  IEEE). 

6.  Use  early  defect  detection  and  defect  prevention. 

7.  Use  operational  definitions  to  define  requirements. 
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Why  Focus  on  Requirements? 

A  Practical  Requirements  Classification 
CMMI  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 
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1.  Define  Lean  Requirements 
Processes  (REQM,  RD) 
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1 


Manage  Requirements  (REQM) 


Purpose:  Effectively  Manage  Requirements  Changes 


Inputs 

Entry 

Tasks 

eXit 

Outputs 

Customer 

Req. 

•  Product 
Req. 

Cust  Req./ 
Prod  Req. 
Inspected 
AND 

Baselined 
AND 
CR/PR’s 
Not  all 

1 .  Perform  CCB 
Meeting  Procedure 

2.  Perform 

Change  Control 
Procedure 

3.  Perform 

Release 

Procedure 

•  CR/PRs 
are 

Resolved 

AND 

Cust  Req./ 
Prod  Req. 
Inspected 
AND 

Under 

•  Customer 
Req. 

•  Product 
Req. 

•  Change 
Requests 

•  Baselines 

•  Problem 
Reports 

Closed 

Best-In-Class 

Metrics 

CM 

•  Releases 

Roles:  Project  Manager  (PM),  CCB 
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1.  Example  Lean  NASA  JPL 
MGSS  CM  Process 


7.1  Perform 

CM  Planning 

7.2  Perform 
Configuration 
Control 

7.3  Perform 
Configuration 

Status 

Accounting 

7.4  Perform 

CM 

Audits 

[Olson  2006a]  Olson,  Timothy  G.,  “Defining  a  Lean  CM  Process  at  NASA  JPL”,  Presentation,  NDL4  CMMI  Conference,  November  2006. 
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1.  Example  Lean  CM  Process 


5  W’s  on 
1  Page 
in  a 

Process 

Model 


Patent 

Pending 

Approach 


7.1  Perform  CM  P  Ian  n  i  ng 

PURPOSE  :  Develop  CM  Plans  th  at  m  e  et  project  needs. 


INPUTS/EN  T  RY  CR  ITERIA  P  ROCES  S 

CONTEXT 

•  Project  initiated  AND  Draft  Project  PI  an 

•  Organiz  a  ti  o  nal  CM  PI  a  n  is  a  pproved  and  m  e  ets  CM  Standard. 


ROLES  &ACTIV  I  TIES: 

Pr  oj  ect 


[Olson  2006a]  Olson,  Timothy  G.,  “Defining  a  Lean  CMProcess  at  NASA  JPL”,  Presentation,  NDL4  CMME Conference,  November  2006. 
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2.  Use  CM  and  CM  Metrics 


Fundamental  Baselines 


Place  the  requirements  under  formal  CM  and  use  CCB’s  to 
control  changes. 

Example  CM  Metrics; 

•  Number  of  CRs/PRs  (e.g.,  open  vs.  closed  over  time) 

•  Requirements  Volatility  (e.g.,  number  of  CRs  per  requirement) 
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3.  Example  Requirement  Metrics 


# 

Requirement 

Reference 

(eg-, 

customer) 

Allocation 

Stability 

(H/M/L) 

Risk 

(H/M/L) 

Priority 

(H/M/L) 

1 

System  shall  send 
an  RTF  FAX 

SOW#  10-20.3 

Software 

H 

L 

M 

2 

Aircraft  position 
shall  be  updated 
by  the  Inertial 
Navigation 

System  (INS) 
Solution 

ORD  #2-30- 
20.3.4.4 

Software 

M 

M 

H 
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4.  Documentation  Framework 


World-Class  Quality 


•  Slide  adapted  from”A  Software  Process  Framework  for  the  SEI  Capability  Maturity  Model”,  CMU/SEI-94-HB-01 
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4.  Requirements  Process  -  NASA 
Onboard  Shuttle  Project 
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5.  IEEE  SyRS  and  SRS 
Standard  Outlines 


SyRS 

1.0  Introduction 

2.0  General  System  Description 
3.0  System  Capabilities,  Conditions, 
and  Constraints 

3.1  Physical 

3.2  System  Performance 
Characteristics 

3.3  System  Security 

3.4  Information  Management 

3.5  System  Operations 

3.6  Policy  and  Regulation 

3.7  System  Life  Cycle 
4.0  System  Interfaces 


SRS 

1.0  Introduction 

2.0  Overall  Description 

3.0  Specific  Requirements 

3.1  External  Interface 
Requirements 

3.2  Functional  Requirements 

3.3  Performance  Requirements 

3.4  Design  Constraints 

3.5  Software  System  Attributes 

3.6  Other  Requirements 
Appendices 

Index 


•  Adapted  from:  IEEE  Std  1233-1998 


•  Adapted  from:  IEEE  Std  830-1998 
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5.  Organizing  SRS  Section  3 


SRS  Section  3  can  be  organized  by: 

•  Mode 

•  User  Class 

•  Object 

•  Feature 

•  Stimulus/Response 

•  Functional  Hierarchy 

•  Multiple  organizations 
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*6.  Example  Requirements  Checklist 

Categories 

1.  Clarity 

2.  Completeness 

3.  Complexity 

4.  Consistency 

5.  Constraints 

6.  Feasibility 

7.  Functionality/Logic 

8.  Interfaces 

9.  Standards 

10.  TBDs 

11.  Testability 

12.  Traceability 
Etc. 
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7.  Example  Operational  Definition 


What  is  a  good  requirement?  When  is  a 
requirement  defined?  Questions  like  these  are 
difficult  to  answer  without  operational  definitions. 

An  operational  definition  precisely  and  concisely 
defines  a  measurable  requirement  that  states 
[NASA  96]: 

•  What  it  has  to  do 

•  How  well  it  has  to  do  it 

•  Under  what  conditions  it  has  to  do  it 
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7.  Example  Operational  Definition 


# 

Requirement 

(What) 

Conditions 

Upper 

Limit 

Lower 

limit 

Base 

Measure 

1 

Report  total  percentage 
of  students  that  passed 
the  first  test  and 
graduated 

Students  that 
pass  first  test  by 
=>  70%  score 

Calculate 
Percentage 
to  3  decimal 
places 

Plus  or 
minus  .001 

Percent 

2 

Report  total  percentage 
of  students  that  failed 
the  second  test  and  did 
not  graduate 

Students  that 
failed  second 
test  by  <  a  70% 
score 

Calculate 
Percentage 
to  3  decimal 
places 

Plus  or 
minus  .001 

Percent 
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Why  Focus  on  Requirements? 

A  Practical  Requirements  Classification 
CMMI  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 


Training  Material  Used  with  Permission  and  Licensed  by  Lean  Solutions  Institute,  Inc.  (LSI) 


Slide  31 


World-Class  Quality 


Some  Advanced  Strategies 


Juran  Model:  Customer  requirements  are  written  in 

the  customer’s  language,  then  translated  into  the 
product  requirements  written  in  producer’s 

language. 


OFD/Juran’s  Quality  Planning  Process:  Measurable 
requirements  that  meet  customer  needs  using  a 
defined  process  (e.g.,  House  of  Quality). 

Usage  Scenarios/Use  Cases/Operational  Scenarios: 

A  powerful  way  to  identify  requirements  based  on 
user  needs. 


Requirements  written  in  formal  languages. 
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Why  Focus  on  Requirements? 

A  Practical  Requirements  Classification 
CMMI  Requirements  Overview 
Practical  Approaches  for  Requirements 
Requirement  Examples 
Some  Advanced  Approaches 
Summary 
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Summary 


The  hardest  single  part  of  building  a  system  is  the 
requirements. 


The  top  requirements  problems  are  inadequate 
requirements  specifications,  changes  to 
requirements,  and  lack  of  user  input. 

Requirements  elicitation  problems  fall  into 
problems  of  scope,  understanding,  and  volatility. 

There  are  practical  strategies  that  you  can  use 
today  that  will  help  you  address  problems  with 
requirements. 
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^The  CAE/Simulation  market  continues  to  see  rapid  and  sustained 
growth 

.SfTwo  recent  innovations  within  the  Analysis  &  Simulation  community  are: 

1 .  The  low  cost  of  compute  capacity 

2.  The  ever  increasing  sophistication  of  simulation  software 

^  The  use  of  stochastics  has  been  validated  in  the  commercial  automotive 
crash  and  test  applications 

■*®The  use  of  stochastics  is  applicable  across  engineering  disciplines 


These  trends  are  continuing  and  we  can  now  expect  to  mimic  true 
lifelike  analysis  through  realistic  and  verifiable  iterative  analysis. 


www.nafems.org 
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NAFEMS  Objectives 

NAFEMS  mission  is  to  act  as  a  trusted  source  and  a  collaborative  resource  for 
the  best  engineering  modeling,  simulation  and  analysis  practices  in  the 
development  of  safe,  reliable,  and  affordable  products.  Its  focus  is  to  champion 
and  improve  best  practices,  to  promote  and  enrich  educational  opportunities 
aligned  with  the  rapidly-advancing  technologies,  and  to  advance  the  productivity 
and  quality  of  virtual  product  development  processes. 

Specific  objectives  of  NAFEMS  are  to: 

-  Promote  COLLABORATION  within  the  international  engineering  analysis  and 
simulation  community, 

-  Stimulate  INNOVATION  via  transfer  of  knowledge  in  the  use  of  advanced  scientific, 
engineering  and  computing  technologies, 

-  Maximize  PRODUCTIVITY  through  improved  best  practices  used  in  product 
development  engineering  processes, 

-  Implant  QUALITY  in  the  methods  and  techniques  exploited  by  virtual  product 
development  processes. 

NAFEMS  is  a  not-for-profit  membership  association  of  nearly  800  companies 
from  all  over  the  world. 
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NAFEMS  Stochastics  Working  Group 


WS\NG  Purpose: 

•  Promote  the  adoption  and  further  development  of 
practical  applications  to  meet  the  Value  Propositions 

•  Give  unique  insight  and  perspective  into  the  area  of 
stochastics. 


•  Collaborate  on  recent  developments 

•  Share  breakthrough  technologies 


www.nafems.org 
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NAFEMS  Stochastics  Working  Group 


i^'Mix  of  industrialists,  consultants,  vendors, 
and  academia: 


•  Provide  recommendations  to  advance  the  user 
community 

•  Share  breakthrough  technology  to  the  dedicated 
community 

•  Provide  support  to  the  SWGSC 

•  Publish  whitepapers 

•  Focus  on  the  user  community 


www.nafems.org 
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NAFEMS  Stochastics  Working  Group" 
Steering  Committee  (SWGSC) 

M  Based  on  a  core  team  of  9  members: 

•  Proactively  represent  the  working  group 

•  Provide  recommendations  to  advance  the  user 
community 

•  Share  breakthrough  technology  to  the  dedicated 
community 

•  Publish  whitepapers  (with  SWG  support) 

End-user  driven 


www.nafems.org 


SWG  Steering  Committee 

Members: 

>  Michel  Klein  -  ESA 

>  Sadek  Rahman  -  Daimler  Chrysler 

>  Tsuyoski  Yasuki  -  Toyota 

>  Alexandar  Karl  -  Rolls-Royce 

>  Raj  Rajagopal  -  Pratt  &  Whitney  Rocketdyne 

>  Kazuhiro  lijima  -  Nissan 

>  Mats  Larsson  -  SAAB 

>  Rodney  Dreisbach  -  Boeing 

>  Mary  Fortier  -  General  Motors 
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Simulation-Supported  Decision 

Making  Gene  Allen 

Director,  Collaborative  Development 
MSC  Software  Corporation 


Simulation  -  A  Tool  for 
Decision  Making 

*  Quickly  Identify  and  Understand  How  a 
Product  Functions: 

•  What  are  the  major  variables  driving 
functionality? 

•  What  are  the  combinations  of  variables  that 
lead  to  problems  in  complex  systems? 

*  Ability  Exists  Today 

•  Due  to  advances  in  compute  capability 

Decision  Maps 
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Generation  of  Decision  Maps 


Decision  Map  -  a  2-D  view  correlating  Results 
generated  from  Monte  Carlo  Analysis 

*  Incorporates  Variability  and  Uncertainty 

*  Updated  Latin  Hypercube  sampling 

*  Independent  of  the  Number  of  Variables 

*  Results  with  100  runs 

*  Does  Not  Violate  Physics 

*  No  assumptions  of  continuity 

*  “Not  elegant,  only  gives  the  right  answers.” 
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Decision  Maps  to  Understand 

Cause  &  Effect 


Dimension  Fraction 


Rib  Spacing 


Mass 


Me*  oiobai  Disp 
1  Local  Disp  Factor 


Peak  Lctalion 


First  Mode 
*X Becinrf  MGdii 
*  *  Sire  4$  Dec 


ess  Irfain  Rib  \ 
Stress  Inner  Flan 
i  s  ress  Outer  f 
*  ^Slness  Rm 


Input 

✓  Variables 


Outer  Riange 


Ranks  input  variables  and 
output  responses 
by  correlation  level 

Follows  MIT-developed 
Design  Structure  Matrix 
model  format 

Filters  Variables  Based 
on  Correlation  Level 


Output 

Variables 
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A  Decision  Map 


Upper  right  - 
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Generation  of  Decision  Maps 
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Enterpriser  Stimulation  Knowledge  Base 


Correlation  Map: 

-  Includes  All  Results 
-  Highlights  Key  Variables 
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Monte  Carlo  Analysis 


Frwuency  Mode  4 


Frequency  Mode  3 
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S»»  Pan  203  ■  Eff  fac  P2 


Sources  of  Variability 

■  Material  Properties 

■  Loads 

■  Boundary  and  initial  conditions 

■  Geometry  imperfections 

■  Assembly  imperfections 

■  Solver 

■  Computer  (round-off,  truncation,  etc.) 

■  Engineer  (choice  of  element  type,  algorithm, 

mesh  band-width,  etc.) 


Solution: 

Establish  tolerances  for  the 
input  and  design  variables. 

Measure  the  system’s 
response  in  statistical  terms. 


The  Fundamental  Problem  .. 
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Structural  Material  Scatter 


MATERIAL 

CHARACTERISTIC 

CV 

Metallic 

Rupture 

8-15% 

Buckling 

14% 

Carbon  Fiber 

Rupture 

10-17% 

Screw,  Rivet,  Welding 

Rupture 

8% 

Bonding 

Adhesive  strength 

12-16% 

Metal/metal 

8-13% 

Honeycomb 

Tension 

16% 

Shear,  compression 

10% 

Face  wrinkling 

8% 

Inserts 

Axial  loading 

12% 

Thermal  protection  (AQ60) 

In-plane  tension 

12-24% 

In-plane  compression 

15-20% 

Source:  Klein,  M.,  Schueller,  G.I.,  et.al., Probabilistic  Approach  to  Structural  Factors  of  Safety  in  Aerospace, 
Proceedings  of  the  CNES  Spacecraft  Structures  and  Mechanical  Testing  Conference,  Paris,  June  1994, 
Cepadues  Edition,  Toulouse,  1994. 


The  Deception  of  Precise  Geometry 

Geometry  imperfections  should  be  described  as  stochastic  fields. 


Monte  Carlo  Results  show  Reality 


Collection 
of  computer 
runs  = 
Simulation 


Single 
computer 
run  = 
Analysis 


Understanding  the  physics  of  a  phenomenon  is  equivalent  to  the 
understanding  of  the  topology  and  structure  of  these  clouds. 


Understanding  MCS  Results 

•  Simulation  generates  a  large  amount  of  data. 

•  A  typical  simulation  run  requires  around  100  solver 
executions. 

•  Each  combination  of  hundreds  to  thousands  of 
variables  produces  a  point  cloud.  In  each  cloud: 

•  POSITION  provides  information  on  PERFORMANCE 

•  SCATTER  represents  QUALITY 

•  SHAPE  represents  ROBUSTNESS 

KEY: 

*  REDUCE  the  Multi-Dimensional  Cloud  to 
EASILY  UNDERSTOOD  INFORMATION 

•  Condense  into  a  DECISION  MAP 

•  Variables  are  sorted  by  the  strength  of  their 
relationship 
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Monte  Carlo  Simulation  Results 

Number  of  2D  Views  of  Results  =  Sum  of  all  integers  from  1  to  (Number  of  Variables  -1) 


PX  DJ3  D.+  □.+□  PJ 


Figure  1.10:  (It*)  and  (right)  vrasus  thickness. 


Figure  1.17:  (frx  (left)  and  ffjtn  (right)  vktsus  thiduiess. 


pjhi  nx  ps  p.+  p.+3  pj  ps  ns  nx  ps  p.+  p.+s  pj  ps 

Ul  Ul 


Figure  1.16:  <TJEtH  (It*)  and  ffJEl  (right)  vusus  thickness. 


12  of  the  78 
2D  views  that 
resulted  from  a 
simulation  with 
6  outputs  from 
a  scan  of  7 
inputs  with 
uniform 
distributions. 


Decision  Maps: 

Understanding  Cause  and  Effect 

*  Displays  condensed  information  from  hundreds  of  analysis  runs. 

*  Decision  Map  =  Structured  Information  =  Knowledge 

*  A  Decision  Map  helps  an  engineer: 

*  Understand  how  a  system  works. 

*  How  information  flows  within  the  system. 

*  how  variables  and  components  correlate. 

*  Make  decisions  on  how  a  design  may  be  improved. 

*  Identify  dominant  design  variables. 

*  Use  as  input  for  stochastic  design  improvement. 

*  Find  the  weak  points  in  a  system. 

*  Find  redundancies  in  a  design. 

*  Identify  rules  that  govern  the  performance  (“if  A  and  B  then  C”). 

There  are  NO  algorithms  to  learn.  The  engineer 
concentrates  on  engineering,  not  on  numerical  analysis. 


Outlier  Identification 
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Most  likely 
behavior 
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7.485 


8.000 


8.515 


9.030 


Outliers:  may 
be  dangerous: 

-  System  Failure 

-  Mission  Failure 


MSCX  Software 


Design  Improvement 

Process 


MSC^  Software* 


APPLICATIONS 


•  Automotive  and  Aerospace  companies  have 
continued  to  expand  use  of  process  since  1997 

•  BMW,  Audi,  Toyota,  Mecedes,  Nissan  and  Jaguar 
have  expanded  Computer  Clusters  for  Stochastic 
Car  Crash  Simulation  taking  10’s  of  pounds  from 
car  model  designs. 

•  Aerospace  companies  applying  to  improve 
aerospace  designs.  Alenia  reduced  weight  of  new 
commercial  airliner  tail  by  6%. 


Courtesy  of  BMW  AG 


Courtesy,  Alenia  Aeronautica 


Process  for  Decision  Support 


•  Model  a  multi-disciplinary  design-analysis 
process 

•  Randomize  the  process  model 

•  Run  Monte  Carlo  simulation  of  the  model 

•  Process  Results 

*  Correlation  Maps  showing  Cause  and  Effect 

*  Outlier  identification  showing  anomalies 

*  Direction  for  Design  Improvement 
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Correlation  Maps  -  Filter  Complexity 

while  Modeling  Reality 

*  Identify  what  influences  functionality 

*  Address  Uncertainty  and  Variation 

*  Provides  credibility  in  modeling  &  simulation 

*  Results  clouds  represent  what  is  possible 

*  Easy  to  use 

*  No  methods  or  algorithms  to  learn 

*  Reduces  risk  through  better  engineering 

*  Takes  all  inputs  into  account  vice  using  initial 
assumptions 

*  Changing  the  general  engineering  process 
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10th  Annual  NDIA  Science  &  Engineering  /  DoD  Tech  Expo 
“Reducing  Technology  Risk  in  Acquisition  Programs” 

Testing  Concept  of  Operations  (C0N0PS1 
In  DoD’s  Net-Centric  Environment 


Mr.  Steve  Reeder 

SCRA 

5300  International  Boulevard,  N.  Charleston,  SC  29410 

steve.reeder@isn-scra.orn 

IP)  757-203-4421,  (F)  043  700-3250 
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GCCS-J  4.x  External  Interface  Architecture 
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DoDs  responsibility  is  the 
management  of — ujJxujjjjj, 


Principles  if  War 


•  Objective 

Clearly  defined,  decisive  and  attainable 
objective.  Each  operation  must  contribute  to  the 
ultimate  strategic  aim. ... 

•  Offensive 

Seize,  retain,  &  exploit  the  common  objectives. 
Means  to  maintain  freedom  of  action  &  achieve 
decisive  results. 

•  Mass 

Synchronizing  all  the  elements  of  combat  power. 
Mass  the  effects  not  necessarily  the  forces. 

•  Economy  of  Force 

No  part  of  the  force  should  ever  be  left  without  a 
purpose 

•  Maneuver 

Movement  of  forces  in  relation  to  the  enemy  to 
gain  positional  advantage.  Continually  pose  new 
problems  for  the  enemy  by  rendering  his  actions 
ineffective  &  eventually  defeating  him. 


J'SCRA 
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Regardless  the 
nt  Technology 


Unity  of  Command 

For  every  objective,  seek  unity  of  command  and  unity 
of  effort.  Unity  of  command  means  that  all  the  forces 
are  under  one  responsible  commander 

Security 

Never  permit  the  enemy  to  acquire  unexpected 
advantage.  Protecting  the  force  increases  friendly 
combat  power.. 

Surprise 

Strike  the  enemy  at  a  time  or  place  or  in  a  manner  for 
which  he  is  unprepared 

Simplicity 

Prepare  clear,  uncomplicated  plans  and  concise 
orders  to  ensure  thorough  understanding 
effectiveness 
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First  and  foremost 

USJFCOM  is  the 


JOINT  CAPABLE  FORCES 


Joint  Warfighter 

Advocate 


.7SCRA 
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FORCE  EMPLOYMENT 

Air,  Land,  and  Sea  Operations 
CAS  Planning  and  Execution 


FORCE  PROJECTION 

Joint  Operation  Planning  SITUATIONAL  AWARENESS 

&  Execution  System  (JOPES) 

ADAPTIVE  PLANNING 


SITUATIONAL 

AWARENESS 

Common  Operational 
Picture  (COP) 


FORCE  READINESS 

Readiness  Assessment  System  (RAS) 

Global  Status  of  Resources  and 
Training  System  (GSORTS) 


INTELLIGENCE 

Integrated  Imagery  Intel  (13) 


FORCE  PROTECTION 


Early  Warning  and  Integrated  Air 
and  Missile  Defense 


J^SCRA 
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Program  Decision  Memorandum  (PDM)  III, 
December  20,  2005,  Tasked  the  Assistant 
SecDef  for  Networks  &  Information 
Integration  /  DoD  Chief  Information  Officer 
(ASD(NII)  /  DoD  CIO.... 

“To  accelerate  the  provisioning  &  adoption 
of  Core  Enterprise  Services  (CES)  across 
DoD. 

In  commercial  industry  speak,  that  means  to 
start  developing  a  System  Oriented 
Architecture  (SOA)  approach  for  C2. 


"'SCR  A 
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The  DoD  must  continue  to  evaluate/assess  technology’s 
impact  on  the  current  war.  And  quickly  adopt  approaches  that 
increase  our  combat  capabilities 


•  Emerging  technologies,  like  SOA  and  innovative 
CONOPS  must  accelerate,  together 

•  Viable  technologies  must  be  rapidly  integrated 
into  current  C2  practices,  allied  operations, 
training,  and  doctrine  for  maximum  effectiveness 

•  Warfighter  needs  are  dynamic,  our  coalition 
arrangements  are  unique,  and  the  “funding- 
requirement-acquisition”  process  is  unacceptable 
in  the  ‘immediate’  for  the  soldier  on  the  patrol 


We  believe  that  Net-Centric  Environment  “e.g.  SOA  approach”  is  the  next  principal 
mechanism  for  enhanced  Command  Capability  of  Joint  C2. 
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Familiar 

Less  familiar 

What  we  use 

FOCUS 

What  we  use  and  how  we  use  it 

Technology  affects  on  system 

SOLUTION 

Technology  +  method  +  people 

capability 

affect  on  operational  capability 

Developers'  perspective 

PERSPECTIVES 

Warfighter  perspective 

Hardware  and  software  must  be 

CENTRAL  RULE  or 

Materiel  and  non-materiel  must  be 

developed  together 

CONCEPT 

developed  together 

SoS  assessment 

APPROACH 

MCP  assessment 

-  OT&E  focus  on  the  system 

-  Holistic  focus  on  all  components 

System  centric 

CENTRICITY 

Capability  centric  (Warrior) 

Focus  on  Joint 
Warfighter’s  urgent 
operational  need  - 
solution  providers  must 
forge  a  single 
1 integrated ’  enterprise 
to  reduce  risk  in 
satisfaction  of  that 
need. 


Changing  the  Business  Model  Requires: 

(1)  Willingness  to  work  together  to  leverage  each  others  core  competencies 

(2)  Focus  on  Joint  Warfighter  as  central  driver  -  solution  need  originator  and  evaluator 

(3)  Commitment  to  providing  meaningful  services  rather  than  inflexible  “products” 


J^SCRA 
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■  TRANSFORMATION  EFFORTS: 

•  Moving  away  from  a  Soviet  based  system 

•  Moving  to  a  professional  as  apposed  to  a  conscript 
based  force 

•  Moving  to  a  capitalistic  based  economic  model 

•  Moving  to  asymmetric  warfare 

•  Moving  to  a  net-centric  combat  capable  force 


CliTON 


At  the  request  of  Poland’s  Chief  of  Defense  (CHOD),  a  combined 
NATO  and  USJFCOM,  Poland’s  Military  staff,  plus  Industry  and 
Academia  constructed  a  near  term  Common  Operating  Picture 
(COP). 

Constructed  a  near  term  SOA  environment  to  integrate  Poland’s  Air, 
Land  and  Sea  into  a  combined  Common  Operational  Picture. 

-  Supported  Poland’s  role  as  a  NATO  member  &  US  strategic  partner 


J^SCRA 
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CHOD& 
Operational 
Command’s 
Critical  Information 
Requirements 


BRITE  interface  incorporated  in  every  system 

□  Automatic  discovery  add-ons 


BRITE  =  Baseline  for  Rapid  Iterative 
Transformational  Experimentation 


J^SCRA 
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Integrate  Existing  and  Emerging 
C2  Capabilities 


Sustained  by  Global  Information  Grid  Enterprise 


Integrate  Soluti 
with  DoD’s 
Net-Centric 
Data  Strateg 


Support 

Enterprise-based 

Joint 

Architecture 


These  web  systems  and  services  will 
have  a  unique  combination  of 
characteristics  that  differentiate  them 
from  more  conventional  legacy  client 
server  applications.  In  particular,  they 
tend  to  include: 

•  Architecture  places  data  at  the 
center  of  its  design:  Enterprise 
Resource  Pattern  (ERP)  & 
Enterprise  Service  Bus  (ESB) 

•  ERP  standardizes  access  to 
any  C2  domain  object  (APIs) 

•  ESB  publishes  messages 
based  on  an  event/trigger 


Services  and  Net-Centric  Enterprise  Services  .  Rapid|y  changing  techno|ogies 

e.g.  more  actors,  platforms, 
networks,  and  services  not 
applications 


J^SCRA 
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CMMMMfl 


Current  C2  Systems 


r 

Graduation 


Increasing 

Capability 

Maturity 


J 

Candidate 

Services 


Net-Enabled  Command 


Capability 


Systems  transformed  into 
discreet  service  capabilities 


C2 

Capabilities 


Systems  Focus 
Stove-piped  systems 
Information  push 


“The  Sandbox” 


Operations  Focus 
Net  &  Data  Centric 
Information  Pull 


Joint  Warfighters  Command  and  Control  Need  Driven 


With  the  net  centric  approach,  user  engagement  occurs  in  the  “ sandbox ” 
during  the  combined  evaluation  referred  to  as  the  “piloting”  events. 


J'SCRA 
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REQUIREMENTS  OPERATIONS 


Capability 

Validsriion 

and 

Deployment 


Testing  & 
Evaluation 


ACQUISITIONS  PERSISTENT  T&E  ENVIRONMENT 


Capability 

Articulation 


System 

Engineering 


J'SCRA 
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Industries  Mixed  Results 


Mixed  Results 

How  has  your  company's  SQA/Web 
services  adoption  performed? 


32% 
Fallen  short 
of  expectations 


0% 

Buie 

expectations 


%  Met 
expectations 


Data:  Inlormation  Wuuk  Research  SOA/Web  services  survey 
ol  278  business  technology  professionals;  229  comnanns 
using  SOA/Wc-b  services 


J'SCRA 
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David  Linthicum 


Linthicum  Group 

Top  5  Mistakes  w/  SOA, 

1.  Not  enough  trained  IT/SOA  architects  to  put  on  the  problem. 

2.  “Manage  by  Magazine”  approach  to  SOA. 

3.  Don’t  understand  the  unique  nature  of  their  problem  domains. 

4.  Treat  SOA  as  a  project,  not  a  journey. 

5.  Unable  to  define  the  value. 


0h>  by  the  wav-  ■  . 

“I  Actively  track,  «n  said’ 
SOA  standards  differen' 

A?^l0;ia?d“P«cative 


any  one  point 


m  time. 


Jim  Green, 

Designing  Reusable  Software, 

-  Types  of  services: 

(1)  put  data  in,  (2)  get  data  out 

-  SOA  &  error  handling  =>  careful  planning 


IT’s  Challenge  —  Deliver  the  Data  with  Flexibility  &  Agility 


Intel 

Systems 


Data 

Services 

Layer 


Intel 

Data 


mm 

t\  .  1 

= 

—  —  tan 

-  s 

Dashboards  Reporting  Applications 


“No  SOA  plan  is  complete  without 
a  data  services  /ayer" 

Source:  AMR  Research 


Constant 

Change 


Virtualizes 

a 

Abstracts 


Siloed 

& 

Complex 

COMPOSITE 

\  \w  IK  i: 


^SCRA 
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CMMM 


Hub  Vandervoort,  CTO,  Progress/Sonic 

His  Key  concept  was  Enterprise  Service  Bus  (ESB) 
Service  Requires  alignment  across  4  dimensions 
Functional,  (2)  Structural  (3)  Behavioral  (4)  Performance 

Interaction  Model  (-Request  Reply,  -Store  &  Forward,  - 
Pub/Sub,  -Bulk  transfers) 


MechfwtoQy 
Cofluihints 

Steve  Kahn,  Bearing  Point 

•  Discussed  two  SOA  projects  (Insurance 
Company  &  Commercial  packaging  firm) 

•  Focus  on  the  business...,  technology  is 
never  enough. 


Some  Final  Thoughts 


■  SOA  Maturity 

■  Incremental  approaches  work  best 

■  Expect  to  get  smarter  along  the  way 

■  Business  Process  Management  and  SOA 

■  BPM  is  the  ultimate  enabler  of  return  on  SOA  investment 

■  BPM  is  to  SOA  what  a  conductor  is  to  an  orchestra 

■  Business  processes  are  built  from  high-level  composite 
servi  ces 

■  Invoke  business  processes  as  services 

■  Knock  down  remaining  impediments 

■  Maintain  Leadership  Support 


SOA  Case  cs 


•  Zda  y  3o.:  -  rq=o  n 


J^SCRA 
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Booz  |  Allen  |  Hamilton 

delivering,  results  tn art  endure 

Melissa  Soley,  BAH,  Trans-National  COP 

■  BAH  Mission  Engineering  (ME)  method  is  a 
bottom-up  IER  data  capture  approach 

Very  intensive  data  capture  approach 

■  Point  of  interest:  80%  of  an  Intel  Analyst’s  time 
is  spent  simply  retrieving  data  not  analyzing. 


High  Level  Operational  Architecture 


OPERATIONAL/STRATEGIC 


QUADRANT 

■  Int ell  Architecture  Integration 

■  System  Development  Compliancy 

■  Program  of  Record  Alignment 

■  Ability  to  Standardize  Impact  DCR 

Materiel  Solutions 


TACTICAL/OPERATIONAL 


QUADRANT 


■  DCGS  JCD  COII OPS  Development 
-  COAL ITIOII  IPT  Support 
■  DDTE  Implementation  Benefits 
■  DCR  Development  Harmony 
■  JIOC  Mi t nation  Process 


J'SCRA 
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AmberPoint 

Sean  Fitts,  Amber  Point 

•  Keys  to  SOA  Runtime  Gov’n 

Visibility  =>  what  is  going  on  &  whoTs 
using  it? 


Control  =>  Actions  to  prevent  or 
correct  issues 


Integrity  =>  Ensuring  changes  don’t 
impact  the  whole  infrastructure 


The  SOA  Validation  Problem 

*  Business  System  Integrity  Always  at  Risk 


Service  reuse  creates  dependencies 

Impact  of  any  changes  ripple  throughout  the  system 

Real  impact  of  planned  changes  is  hard  to  predict 

Impact  of  unplanned  or  unannounced  changes  can  be  devastating 

Yet,  it  quickly  becomes  impossible  to  setup  and  replicate  all  dependent 
systems  fior  testing  elsewhere 

Need  way  to  continuously  check  for  integrity  -  both 
in  staging  and  in  production 

Amber  Pqint  14 
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fMjTjTTIgjjl^r 


Universal 

Joint 

Task 

Lists 

(UJTLs)  ^ 


Governance 


Test 

Threads 


lanagement 


Master 

Training  — _____ 
Guide  ~ 

(MTG) 

Other 

Authoritative 

Sources 

Reporting 


lecurity 


service 

Discovery 

Mediation 


Messaging 


anc 
Rqmts 


Operational 

Community 


Operational  Requirements 

Subject  Matter  Experts  (SME) 


Materiel  Developer 

Subject  Matter  Experts  (SME) 


System 
Community 


So  what  did  he  sayP 


DoD’s  C2  environment  has 
@  7  million  customers 

Our  business  is  the 
management  of  violence 

JFCOM  is  the  Joint 
Warfighter  Advocate 

DoD  is  moving  to  Net 
Centric  C2 

DoD  will  continue  to  adapt 
to  change 


Poland’s  military 
transformation  & 
movement  toward  Net 
Centricity 

NECC  programmatic 
processes 

Industries  views 

NECC  testing  concept 

DoD  is  in  the  early 
stages  of  SOA  adoption 


J^SCRA 
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BACKUP  SLIDES 


Use  an  Operational  Mission  Thread  Concept 


SHARED  SITUATIONAL  AWARENESS 


Operational  Event  Trace 
Description  (OV-6c) 


Event  Table 


J'SCRA 
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Integrated  Risk  and  Knowledge 
Management  for  Exploration 


David  M.  Lengyel 

Risk  and  Knowledge  Management  Officer 
Exploration  Systems  Mission  Directorate 
NASA  Headquarters 


Introduction 


Objective: 

Customer: 

Goal: 


Introduce  and  discuss  real  work  process  improvements 
that  utilize  organizational  management  innovations  and 
leverage  existing  ESMD  information  technology  resources 

The  ESMD  civil  servants  and  contractor  work  force 

No  nonsense,  straight-up,  “Real  Deal”  approaches  to 
make  your  job  more  fun  and  make  you  more  effective 

-  Work  more  effectively  and  efficiently 

-  Make  better  -  more  risk  informed  decisions 

-  Manage  risks  in  a  proactive  fashion 


Not  another  burdensome  management  /  administrative  demand 

on  your  time . This  stuff  will  save  you  time  ! 


Why  Integrate  Risk  and  Knowledge  Management? 


Designing  a  complex  architecture  of  hardware,  software,  ground  and 
space-based  assets  to  return  to  the  Moon  and  then  go  on  to  Mars  will 
require: 

1 )  an  effective  strategy  to  learn  from  past  lessons,  and 

2)  a  set  of  inter-related  practices  to  generate  and  share  knowledge  for  reuse 
as  we  progress  forward.  ESMD  risk  and  knowledge  management 
communities  have  embarked  on  an  effort  to  integrate  risk  and  knowledge 
management  (KM)  over  the  lifecycle  of  the  Constellation  and  Advanced 
Capabilities  Programs  using  a  set  of  inter-related  strategies,  which 
include: 


Practice  1 :  Establish  Pause  and  Learn  Processes 

Practice  2:  Generate  and  Infuse  Knowledge-Based  Risks  (KBRs) 

Practice  3:  Establish  Communities  of  Practice  (CoP) 

Practice  4:  Provide  Knowledge  Sharing  Forums 


Practice  5:  Promote  Experienced-Based  Training 


ESMD  and  Stealth  KM 


“Knowledge-enabling  processes  (i.e.  process  improvement)  will 
lay  a  solid  KM  foundation  for  future  organizational  evolution  and 
help  align  KM  with  business-based  goals  and  objectives 

Improving  processes  also  provides  an  opportunity  to  deploy 
supporting  KM  tools  and  techniques  such  as  collaboration  or 
CRM  software  and  processes  -  this  can  give  important  momentum 
to  knowledge  workers,  and  can  help  them  to  work  in  a  more 
holistic  and  community-based  way 

Bottom-line:  Process  evolution  equals  culture  evolution” 


Niall  Sinclair 
Author  of  Stealth  KM 


dlengyel@hq.nasa.gov 


4 


Practice  1:  Pause  and  Learn 


PaL  is  modeled  after  the  Army  After 
Action  Review  (AAR)  system  by 
Dr.  Ed  Rogers  KM  Architect  at  the 
GSFC. 

The  idea  is  to  create  a  learning  event 
at  the  end  of  selected  critical  events  in 
the  life  of  a  project.  End  of  project 
reflections  are  good  but  are  too 
infrequent  for  the  organization  to  learn 
in  a  timely  manner. 

PaL  meetings  are  intended  to  be 
integrated  into  the  project  life  cycle  at 
key  points  as  a  natural  part  of  the 
process.  PaL  meetings 
are  structured  and  facilitated  by 
specialists  who  are  not  project 
members 


dlengyel@hq.nasa.gov 
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Attributes  of  a  PaL 


Informal,  facilitated  roundtable  discussion  (1/2  hour  to  full  day) 

-  Includes  moderator  and  rapporteur 

-  Focuses  on  tasks  and  goals  that  were  to  be  accomplished 

Not  for  attribution 

-  Does  not  judge  success  or  failure  (not  a  critique) 

-  Encourage  employees  to  surface  lessons 

Focused  on  particular  area  of  project  life  (phase  and  function) 

-  Management  PaL,  Technical  PaL,  Conceptual  PaL,  et.  al. 

-  Team  participation  may  vary,  depending  on  PaL  focus  and  objective 

Maximizes  participation 

-  Primary  benefactors  are  the  participants  themselves 

-  More  project  activity  can  be  recalled  and  more  lessons  shared 

Must  be  conducted  mside  a  projects  schedule,  not  outside  or  later 

-  Recall  of  key  details  more  likely  and  insights  can  be  immediately 
applied 

-  Affirms  learning  as  integral  part  of  project  life  cycle 


dlengyel@hq.nasa.gov 
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PaL  as  a  Process 


Step  1 

-  Identify  when  PaLs  will  occur 

-  Determine  who  will  attend  PaLs 

-  Select  Moderators,  Rapporteurs 

-  Select  potential  PAL  sites 

-  Review  the  PAL  plan 

Step  2 

-  Review  what  was  supposed  to  happen 

-  Establish  what  happened  (esp.  dissenting  points  of  view) 

-  Determine  what  was  right  or  wrong  with  what  happened 

-  Determine  how  the  task  should  be  done  differently  next  time 

Step  3 

-  Review  objectives,  tasks,  and  common  procedures 

-  Identify  key  events 

-  Rapporteurs  collect  ALL  observations 

-  Organize  observations  (identify  key  discussion  or  teaching  points) 


1  Adapted  from  United  States  Army  Manual:  A  Leader’s  Guide  To  After  Action  Reviews 
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Practice  2:  Knowledge-Based  Risks 


Definition 


Knowledge-Based  Risk  n.  1 .  A  risk  based 
on  lessons  learned  from  previous  experience. 
2.  A  closed  risk  with  documented  lessons 
learned  appended.  3.  A  means  of 
transferring  knowledge  in  a  risk  context. 


dlengyel@hq.nasa.gov 
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Lessons  Learned  on  Lessons  Learned 


*  Start  Early 

*  Need  to  Capture,  Learn  From  and  Repeat  Successes-Need  to  Learn  from  and 
Prevent  Failures,  Mishaps,  Near  Misses 

*  There  was  a  limited  number  of  useful  lessons  learned  in  the  NASA  Lessons 
Learned  Information  System  database.  The  good  ones  are  masked  by  the 
hundreds  of  poor  ones,  so  that  extensive  effort  is  required  to  sort  them  out. 

*  Lesson  Learned  -  Well-understood  mechanisms  for  “transfer  of  knowledge” 
during  Program  development  are  crucial  to  a  successful  long-term  Program. 

*  Flow  all  applicable  Lessons  Learned  into  Requirements,  Processes,  and  Plans. 
Institutionalize  the  Use  of  Lessons  Learned. 

*  Provide  Sufficient  Resources,  Planning,  and  Management  Support  to  Analyze 
and  Incorporate  Lessons  Learned.  NASA  and  Contractor  Must  Work  Together 

*  The  best  lessons  learned  for  running  a  major  program  should  be  captured  in  a 
living  handbook  ofbest  /practices .  New  lessons  learned  should  be  screened  for 
applicability,  and  included  in  the  handbook. 


ESMD  Is  Taking  a  New  Approach  to  Lessons  Learned 


Knowledge-Based  Risks  Strategy 


The  ESMD  KBR  strategy  is  intended  to  convey  risk- 
related  lessons  learned  and  best  practices  to  ESMD 
personnel.  This  strategy  integrates  the  existing 
Continuous  Risk  Management  (CRM)  paradigm  used 
at  NASA  with  knowledge  management--with  the 
primary  focus  on  integrating  transfer  of  knowledge 
through  existing  work  processes  and  not  adding  an 
additional  burden  to  the  workforce  to  incorporate 
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KBR  Process  Flow  Chart 
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KBR  Criteria 


Risks  that  are  "Candidate  KBRs"  should  meet  several  of  the 
following  criteria  (listed  in  order  of  importance): 

(1 )  Were  mitigated  (not  accepted  or  watched) 

(2)  Will  likely  appear  again  in  other  programs  /  projects 

(3)  Included  a  particularly  effective  mitigation  approach  / 
implementation,  or  an  error  in  mitigation  planning  or 
implementation  could  have  been  avoided 

(4)  Was  on  the  performing  organization's  Top  Risk  List  at  some 
point  during  the  life  cycle 

(5)  Was  owned  (and/or  worked  on)  by  a  particularly  knowledgeable 
person  who  could  serve  as  a  "expert"  on  the  risk  topic 
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Application  of  Risk  Management  Assurance  Mapping 


Multiple  KBR  Capture  Points  -  Multiple  Delivery  Points 
Internal  and  External  to  ESMD 
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Number  of  Documented  Risks  Number  of  Documented  Risks 


Knowledge-Based  Risks  (Continued) 


Standalone  Program  Risks 
in  ARM 


More  access  to  risk  information  is 
required  to  dose  “knowledge  gaps ” 

KBRs  will  become  a  living  reference 
over  time  as  risks  are  identified, 
mitigated  and  closed 


Ares-1 


Orion 


Ground  Ops 


Ground  Ops 


Ares-1 

Orion 


KBRs 


Ares-1 


Orion 
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Knowledge-Based  Risks  (Continued) 


ACTIVE  RISK  MANAGER 

RISK  REVIEW 


File  Edit  New  View  Link  Analysis  Reports  Tools  Help 


*  IS) 

□  ! 

ID 

& 

0Q  ARMD 
0ft  ESMD 
OQ  SMD 
3  D  SOMD 

-  Cn  Knowledge  Based  Risks 

-  Q  Program  &  Project  Management  -  01 
O  O  Program  Management 
O  O  Project  Management 
G  Q  Systems  Engineering  -  02 
O  Q  Safety  &  Mission  Assurance  -  03 
gq  Science  /  Technology  -  04 
O  Q  Payloads  -  05 
Q  Q  Aircraft  &  Spacecraft  -  06 
O  O  Mission  Operations  -  07 
O  O  Launch  Vehicle  /  Services  -  08 
OQ  Ground  Systems  -  09 
O  Q  Systems  Integration  &  Testing  - 1 0 
G  Q  Education  &  Public  Outreach  - 1 1 


NASA  Standard  WBS 


ARM  allows  automated  delivery  of  new  KBRs 


dlengyel@hq.nasa.gov 
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Knowledge-Based  Risks  (Continued) 


•  Embedded  3-8  min 
Video  Nugget  with 
Transcript 

•  Related  Knowledge 
Bundles 

•  Related  Content  - 
reports,  documents,  etc. 

•  Threaded  discussion 
(blog)  feature  to  be 
added  to  comment  on 
each  KBR 

•  Hosted  on  ESMD  R&KM 
portal 
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First  Closed  Risk  KBR  -  Lunar  Recon  Orbiter 


LRO  Spacecraft  Delta  II  Booster 


Atlas  V  Booster  LCROSS 

Spacecraft 


*  The  design  of  the  LRO  propulsion  tanks  was  influenced  by  a  number  of 
factors  including  launch  vehicle  characteristics.  The  Delta  II  Expendable 
Launch  Vehicle’s  (ELV)  spin  stabilized  upper  stage  made  the  Nutation  Time 
Constant  (NTC)  a  key  parameter  in  assessing  the  stability  of  the  spacecraft. 
The  uncertainty  in  predicting  the  effects  of  liquid  propellant  motions  and  the 
relatively  large  propellant  load  and  mass  fraction  for  the  LRO  tank  resulted  in 
the  identification  of  a  potential  risk.  Close  coordination  and  communication 
with  all  levels  of  management  early  in  the  design  trade  study  process  allowed 
for  the  effective  mitigation  of  the  risk  and  provided  additional  lunar 
exploration  opportunity. 


dlengyel@hq.nasa.gov 
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Practice  3:  Communities  of  Practice 


Knowledge  resides  with  people  and  is  often  lost  via 
actions  like: 

•  Downsizing 

•  Retirements 

•  Shuttle  Transition 

•  People  Movement 

Participation  in  a  CoP  should  be  considered  part  of 
any  professional’s  career  growth 
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Communities  of  Practice  (Continued) 


“Communities  of  Practice  (CoP)  are  groups  of  people  who  share 
a  concern,  a  set  of  problems,  or  a  passion  about  a  topic,  and  who 
deepen  their  knowledge  and  expertise  in  this  area  by  interacting 
on  an  ongoing  basis” 

“CoPs  share  information,  insight  and  advice.  They  help  each  other 
solve  problems.” 

“They  may  create  tools,  standards,  generic  designs,  manuals,  and 
other  documents — ” 

“Cultivating  CoP  in  strategic  areas  is  a  practical  way  to  manage 
knowledge  as  an  asset,  just  as  systematically  as  companies 
manage  other  critical  assets.” 

Communities  of  Practice.  Wenger,  et  al 
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IT  Enabling  ESMD  CoPs  in  a  Secure  Environment 


The  Confluence  Wiki 

provides  secure 
collaborative 
functionality  within  the 
ESMD  Integrated 
Collaborative 
Environment  (ICE). 
ESMD  Wiki  spaces  now 
number  over  130 


The  PBMA  toolkit 

provides  NASA  CoPs 
with  a  secure 
environment  to  share 
documents,  conduct 
threaded  discussions, 
polls,  manage 
calendars,  locate 
expertise,  collaborate 
and  learn.  Over  30 
ESMD  CoPs  are 
serviced  by  PBMA. 
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Practice  4:  Knowledge  Sharing  Forums 


ESMD  Alumni  Sharing  Events: 

*  These  events  bring  in  alumni  from  Apollo,  Space  Shuttle,  and  other 
programs  to  discuss  their  experiences  and  lessons  learned 

*  This  is  an  extensive,  under-utilized  knowledge  base 

*  ESMD  has  invited  selected  alumni  to  brown  bag  lunches  and  other 
lessons  learned  forums 

Knowledge  Sharing  Workshops  and  Seminars: 

*  At  Knowledge  Sharing  Workshops,  senior  project  leaders  share  their 
insights,  what  they  learned  and  what  they  might  have  done  differently 
based  on  a  recent  project  experience. 

*  These  workshops  are  attended  by  emerging  project  leaders  who  want 
to  understand  the  wisdom  of  successful  project  managers 

APPEL  Master’s  Forums: 

*  Conducted  twice  annually 

*  ESMD  has  and  will  continue  to  participate  in  these  events 


dlengyel@hq.nasa.gov 
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Practice  5:  Experienced-Based  Training 


Project  Management  and  Engineering  Training 

*  Already  conducted  by  APPEL  and  NESC  Academy 

*  ESMD  will  focus  its  efforts  in  training  on  leveraging  the  existing 
infrastructure  of  training  courses  throughout  NASA 

*  ESMD  will  help  shape  existing  courses  by  providing  ESMD-related 
experiences,  gleaned  from  case  studies,  KBRs,  and  other  sources  of 
lessons 

Case  Studies 

*  ESMD  will  facilitate  the  development  of  case  studies  that  will  help 
transfer  the  context  of  program/project  decisions  to  the  workforce  and 
emerging  leaders 

*  Senior  ESMD  managers  would  help  shape  the  content  based  on  their 
experiences  and  leadership 

*  Case  studies  will  make  existing  training  programs  more  relevant  and 
useful  to  upcoming  ESMD  leaders  who  participate 


dlengyel@hq.nasa.gov 
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KM  Practices  and  Tool  Integration 


ESMD  Risk  &  Knowledge  Management  Workgroup 
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ESMD  Risk  &  KM  Teaming 


ESMD  is  teamed  with: 


•  Space  Operations  Mission  Directorate 

•  Office  of  Safety  &  Mission  Assurance 

•  NASA  HQ  Institutions  &  Administration 

•  Academy  of  Program  /  Project  &  Engineering  Leadership 

•  NASA  Engineering  &  Safety  Center  (NESC)  Academy 

•  JSC  Chief  Knowledge  Officer 

•  GSFC  Chief  Knowledge  Officer 

•  MSFC  /  Ares  Chief  Knowledge  Officer 

•  Constellation  Program 

•  ISS  Program 

•  SSP  Program 

•  Pratt-Whitney-Rocketdyne  Chief  Knowledge  Officer 

•  Lockheed-Martin 

•  ATK-Thiokal 

•  United  Space  Alliance,  Office  of  the  Chief  Engineer 

•  The  Aerospace  Corporation 

•  NASA  Alumni  Association 

•  Defense  Acquisition  University  -  Best  Practices  Clearinghouse 


dlengyel@hq.nasa.gov 
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Summary 


“ESMD  faces  exciting  opportunities  and  formidable  challenges.  To 
reduce  risk  and  apply  knowledge  more  effectively,  ESMD  should 
integrate  its  KM,  RM  and  OL  initiatives  into  a  comprehensive  plan 
that  will  accomplish  more  with  less  bureaucracy.  The  goal  is  not 
compliance  with  detailed  processes  and  procedures  but 
compliance  with  intent:  the  intent  to  learn,  to  share  and  probe 
every  possible  angle  so  ESMD’s  missions  have  the  highest 
possible  chance  of  success.  ESMD  must  take  risks  with  ‘eyes 
wide  open’  and  ‘minds  fully  engaged’  at  every  decision,  every 
trade  and  with  every  residual  risk.”  /\  _ 


From:  Strategy  for  Exploration  Systems  Mission  Directorate 
Integrated  Risk  Management,  Knowledge  Management 
and  Organizational  Learning  Whitepaper 
Dave  Lengyel  &  Dr.  Ed  Rogers 


dlengyel@hq.nasa.gov 
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Questions? 


Contact  Information: 

dlenqvel@hq.nasa.qov 

Office:  (202)  358-0391 
Cell:  (202)  253-1762 


dlengyel@hq.nasa.gov 
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CMMI — Update  and  Next  Steps 


NDIA  Systems  Engineering  Conference 

24  October  2007 

Ms.  Kristen  Baldwin 

OUSD(AT&L)  Deputy  Director, 

Software  Engineering  and  Systems  Assurance 


Outline 


•  CMMI-DEV  Guidebook  for  Acquirers 

•  CMMI  for  Acquisition  (CMM-ACQ) 

•  CMMI  Next  Steps  Beyond  vl  .2 

•  CMMI  Constellations,  Focus  Topics  and 
Moving  Forward 


CMMI:  Implementation  Issues 


•  Developers  execute  at  lower  maturity  levels  than  their 
organizations  have  achieved  and  advertised 

•  Assurance  that  new  projects  will  incorporate  CMMI 
processes 

•  Appraiser  quality  -  training,  consistency 

•  Lack  of  agreement  on  what  constitutes  Levels  4  and  5 

-  Requirements  for  demonstrated  behavior 

-  Definition  of  Levels  4  and  5  themselves 

•  Appraisal  disclosure  statement  content 

-  Coverage  of  the  organization  appraised 

-  Performance  on  individual  process  areas 

•  Training  and  education  for  acquirers 

•  CMMI  misuse  in  source  selection 


Proper  use  of  CMMI  requires  knowledge  of  these  issues 


Understanding  and  Leveraging  a 
Supplier’s  CMMI  Efforts: 

A  Guidebook  for  Acquirers 
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CMMI  Acquirer’s  Guidebook 


•  Designed  to  help  an  acquirer  benefit  from  a  supplier’s  use 
of  CMMI-DEV  while  avoiding  the  pitfalls  associated  with 
unrealistic  expectations  related  to  CMMI  level  ratings 

•  Readable  (small)  40  pages  for  the  Program  Manager 

-  Available  at 

http://www.sei.cmu.edu/publications/documents/07.reports/07tr00 

4.html 

•  Part  of  the  CMMI  Product  Suite 

-  Change  requests  and  comments  can  be  submitted  to  cmmi- 

comments@sei.cmu.edu. 

-  Will  be  updated  with  learning  and  experience 

•  Will  be  made  into  a  Continuous  Learning  Module  for  acquirer 
training  with  the  Defense  Acquisition  University 
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Key  Tips  in  the  Guidebook 


•  Do  not  ask  for  CMMI  maturity  levels  in  RFPs 

-  Ask  for  capability  in  processes  that  are  key  to  the  success  of 
your  program 

•  Read  the  Appraisal  Disclosure  Statement  (ADS) 

-  Determine  what  part  of  the  organization  was  actually  appraised 
and  how  it  relates  to  your  program 

-  For  high  maturity  (levels  4  and  5),  determine  what  processes 
were  actually  improved 

-  Ask  for  clarification,  appraisal  findings  if  needed 

•  Recognize  that  levels  are  a  result  of  appraisals  that  cost 
money 

-  Can  achieve  results  using  other  assessment  techniques 

-  Can  do  post-award  checks  to  ensure  your  project  is 
implementing  its  promised  processes 


High  capability  and  maturity  level  ratings 
do  not  of  themselves  guarantee  program  success 
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Guidebook  Bottom  Line 


•  DoD  does  not  place  significant  emphasis  on 
capability  level  or  maturity  level  ratings 

-  Promotes  CMMI  as  a  tool  for  internal  process 
improvement 

•  Lack  of  emphasis  on  ratings  is  prudent 

-  Findings  that  not  all  suppliers  are  exhibiting  behavior 
consistent  with  their  attained  CMMI  maturity  level  rating 

•  Essential  that  DoD  and  industry  use  CMMI  capability 
in  the  right  manner,  with  appropriate  measure,  in 
order  to  realize  benefits 

-  CMMI-DEV  provides  a  set  of  best  practices  to  be 
employed  by  the  supplier 
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CM  Ml  for  Acquisition 
1  Nov  07  release 


CMMI-ACQ 

Development  Strategy 

•  General  Motors  and  the  SEI  developed  the  initial  draft  model 

-  Source  models  included  CMMI  Acquisition  Module  (CMMI-AM)  and 
Software  Acquisition  Capability  Maturity  Model  (SA-CMM) 

-  Incorporated  lessons  from  several  acquisition  organizations  to  adapt 
the  CMMI-DEV  to  their  organization 

-  Pilots  from  several  acquisition  organizations  (DHS,  GAO,  Army,  GM, 
others) 

•  Model  Team  dispositioned  over  700  change  requests  from  stakeholder 
review  and  workshop  to  develop  and  peer  review  recommended 
changes  to  initial  draft 

•  Advisory  Board  of  government  and  industry  stakeholders  established  as 
change  control  board 

•  v0.9  piloted  at  one  defense  agency  and  one  commercial  company 

•  Steering  Group  endorsed  final  product  as  part  of  the  vl  .2  product  suite 

•  Will  be  published  on  1  November,  available  at 

http://www.sei.cmu.edu/cmmi/models/index.html 
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CMMI-ACQ  Development 
Challenges 


•  Model  had  to  explicitly  apply  to  the  acquisition  of 
both  products  and  services 

-  From  IT  outsourcing  to  DoD  acquisition  of  a  weapon 
system 

-Applicable  internationally-recognized  references  and 
glossary  terms  added,  e.g.,  service  level 
measurement 

•  Model  had  to  apply  to  spectrum  of  acquisition 
organizations  from  commercial  industry  to 
government  agencies,  both  large  and  small 
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CMMI-ACQ  vl  .2 

Acquisition  Category  Process  Areas 


Acquisition  Specific-Practice 
Enhancements  to  CMF  PAs 


•  Measurement  and  Analysis 

-  Includes  earned  value  management  material 

-  Consistency  across  the  model  in  measurement  terms 

•  Project  Planning 

-  Includes  establishment  and  maintenance  of  a  project’s 
acquisition  strategy 

•  Project  Planning  and  Project  Monitoring  and  Control 

-  Includes  important  specific  practices  on  transition  to  operations 
and  support 

•  Integrated  Project  Management  and  Organizational  Process 
Development 

-  Includes  material  on  integrated  teaming 

-  Crucial  to  stakeholder  involvement  for  acquisitions  in  a  system  of 
system  environment 
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Highlights  of  Acquisition  PAs 


•  Solicitation  and  Supplier  Agreement  Development 
(SSAD)  and  Agreement  Management  (AM) 

-  Similar  to  Supplier  Agreement  Management  in  CMMI-DEV  but 
greatly  expanded  into  2  PAs 

-  Covers  both  legal  contracts  and  other  forms  of  supplier 
agreements  such  as  interagency  MOAs 

•  Acquisition  Requirements  Development 

-  Similar  to  Requirements  Development  in  CMMI-DEV,  but 
develops  customer  and  contractual  requirements 

-  At  maturity  level  2  due  to  its  importance  in  acquisition 


13 


Highlights  of  Acquisition  PAs 


•  Acquisition  Technical  Management 

-  Emphasizes  technical  reviews  and  technical  performance 
measurement  for  oversight  of  the  supplier 

-  Interface  Management  included  to  complement  the  other  kinds  of 
technical  management  process  areas  (e.g.,  Risk  Management, 
Requirements  Management) 

•  Acquisition  Verification  and  Acquisition  Validation 

-  Similar  to  CMMI-DEV  Verification  and  Validation  PAs  but 
enhanced  for  the  acquirer 
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CMMI  Next  Steps: 
Beyond  vl.2 


Questions  for  v2.0  of  the  Models 
and  Appraisal  Method 


•  Do  we  need  something  different  or  additional  to  define  High 
Maturity  (i.e.  CMMI  Level  4  &  5)? 

•  How  can  we  apply  Lean  techniques  to  CMMI  models?  Appraisal 
methods? 

•  Can  we  eliminate  the  Staged  representation? 

•  Is  the  CMMI  vl  .2  Constellation  Strategy  the  right  approach? 

•  Can  we  identify  “next-generation”  process  improvement 
methodology? 

•  Can  CMMI  be  harmonized  with  other  continuous  process 
improvement  efforts? 

•  Can  repeatability,  consistency  and  overall  model  and  appraisal 
methodology  be  improved? 

•  Are  there  “breakthrough”  concepts  that  we  can  apply  to  overall 
process  improvement? 
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Excerpts  from  Next  Gen  PI 

Workshops 


•  Leaning  the  model 

-  Can  we  lean  for  small  projects?  Can  the  model  have  some  scalability 
according  to  various  factors  (e.g.,  project  size,  PoP,  organization  size)? 

-  Consider  options  for  packaging  (remove  redundancy  or  repackage) 

-  Consider  fundamental,  intermediate  and  advanced  volumes 

-  Consider  architectural  views  for  appropriate  for  the  different  using 
communities 

•  Levels  4-5 

-  Combine  levels  4  and  5  into  one  level  because  of  their  close  tie 

-  4  and  5  are  not  adequately  elaborated  for  implementation  -  may  need 
more  detail  to  drive  proper  behavior 

-  Consider  maturity  levels  within  PAs  (e.g.,  project  management  PAs  for 
each  level) 

•  Constellations  -  the  right  approach? 

-  Alternative  approach:  Start  with  a  CMMI  Model  Framework  (CMF)  and 
add  where  you  need  to,  expand  scope  (+  concept) 

-  Instead  of  creating  constellations,  encourage  projects  to  do  what  makes 
sense  with  respect  to  what  they  are  doing  using  the  parent  model  1 


Excerpts,  continued 


•  Next  Gen  PI  ideas 

-  Consider  better  interfacing  approaches  with  other  methodologies  (e.g., 
six  sigma  for  high  maturity) 

-  Consider  how  CMMI  could  interface  with  other  process  improvement 
methodologies  (e.g.  Lean,  PMBOK,  theory  of  constraints,  next 
generation  IDEAL) 

-  Consider  an  emphasis  on  process  performance  effectiveness  and 
efficiency,  (e.g.,  effectiveness  6  sigma,  efficiency  LEAN) 

•  Leaning  Appraisals 

-  Consider  notion  of  visits  or  interim  steps  (like  ISO  surveillance  audits) 

-  Focus  on  correlation  between  results  and  performance  (process 
reviews) 

-  Make  some  assumptions  that  some  processes  are  in  place  (e.g., 
assume  project  planning  has  happened,  but  don’t  look  at  PP  specifically 
unless  you  see  something  out  of  place  in  PMC;  similarly,  could  start  with 
IPM  for  a  level  3,  or  QPM  for  a  level  4) 
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Next  Steps: 

CMMI  Constellations  and 
Focus  Topics 


Dealing  with  Two  Constellations  in 

the  Product  Suite 


•  The  following  questions  need  to  be  considered 

-  How  does  an  organization  that  does  both  development  and 
acquisition  use  both  models  effectively? 

-  How  does  an  organization  that  uses  both  models  have  efficient 
appraisals? 

-  How  to  keep  the  CMF  consistent 

•  CMMI-ACQ  identified  changes  needed  in  the  CMF  shared  material 

•  There  is  now  a  mismatch  with  CMMI-DEV  vl  .2 

-  How  to  ensure  appraiser  and  instructor  qualifications  for  the  new 
model? 

-  How  do  we  accomplish  training? 
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CMMI  for  the  Service  Sector: 
Some  Questions  to  be  Addressed 


•  What  is  the  requirement/problem  to  be  solved? 

•  What  distinguishes  CMMI-SVC  from  CMMI-DEV  and 
ACQ?  Other  process  models? 

•  What  are  the  characteristics  of  Service  providers? 

•  Is  there  known  benefit  from  Service-specific  process 
improvement?  From  Service-specific  practices? 

•  Can  the  broad  spectrum  of  Services  be  governed  by  a 
single  model? 

•  How  should  Service  Sector  needs  be  incorporated  into 
the  CMMI  product  suite? 


We  are  currently  evaluating  these  questions 

- zn 


CMMI  Focus  Topics 
Business  Rules 


What  is  a  Focus  Topic? 

•  Focus  Topics  provide  additional  guidance  for  the  development  of  CMMI-based 
internal  processes  within  an  area  of  interest 

•  Examples  of  Focus  Topics:  SoS,  Safety,  Security,  COTS 

Business  Rules  for  Focus  Topics: 

•  They  provide  a  “thread”  through  existing  process  areas  to  augment  or  highlight 
a  specialty  area  of  importance  to  an  acquirer  or  developer 

•  They  do  not  introduce  new  process  areas  or  specific  goals 

•  Documented  as  Technical  Notes  (TNs) 

•  Appraisals  shall  not  include  reference  to  Focus  Topics  as  part  of  the  appraisal 
ratings 

Progress  against  Focus  Topics  can  be  included  in  appraisal  findings  for  the  purpose 
of  identifying  strengths  and  weaknesses. 

•  Shall  adhere  to  the  CMMI  Architecture  Document 

•  Steering  Group  and  Sponsors  informed  of  the  possible  Focus  Topic  TN  and  its 
proposed  development  plan  before  work  is  begun  by  the  SEI 

•  SEI  publishes  the  TN  after  a  suitable  set  of  reviews  have  been  completed  and 
comments  have  been  dispositioned  and  accepted 

Ensure  all  parts  of  the  product  suite  are  consistent  and  managed 


Moving  Forward 


•  Evaluate  changes  to  the  CMMI  vl  .2  product  suite  to 
ensure  improvement  goals  are  really  being  met 

-  Integrity  of  appraisals 

-  Quality  of  the  product  suite 

-  Education  of  acquirers 

-  Opportunities  for  streamlining  where  appropriate 

•  Re-look  levels  4  and  5 

-  Consistent  definition  and  appraisal 

-  Relationship  to  other  models  (e.g.  6  sigma) 

-  Appraiser  and  implementer  training  and  understanding 

•  Monitor  Cost  Impacts  and  Return  on  Investment 

-  All  changes  to  the  suite  have  impacts  on  industry  and 
government,  direct  and  indirect 

-  Need  cost  impact  data  from  you!! 
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Questions/Comments? 


Guidebook: 

http://www.sei.cmu.edu/publications/documents/07.reports/07tr004.html 

CMMI-ACQ  Model: 

http://www.sei.cmu.edu/cmmi/models/index.html 

CMMI-AM  Module: 

http://www.sei.cmu.edu/publications/documents/05.reports/05tr01 1  .html 

Ideas  for  Next  Gen  PI: 

Comment  forms  available  on  SEI  website 


BACKUP 


Example:  Published  maturity  levels 
may  be  based  on  a  single  location 
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CMMI-ACQ  Plan  for  V2.0 


•  VI  .2  concentrated  on  the  project-,  or  program-level 
acquisition  best  practices 

•  V2.0  will  add  more  of  the  enterprise/  organization 
level  best  practices  for  acquisition 

-  Address  enterprise  level  acquisition  strategies 

•  Preferred  supplier  strategies 

-  Address  the  Program  Executive  Office  level 

•  V2.0  will  also  benefit  from  change  requests  issued 
from  lessons  learned  using  the  model  globally 
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System  Engineering  in  a 
System  of  Systems  Environment 

A  Defense  Update 

Dr.  J  udith  Dahmann 
The  MITRE  Corporation 

Kristen  Baldwin 
OUSD  (A&T)  SSE/SSA 

NDI A  SE  Conference 
October  2007 


System  of  Systems: 

A  set  or  arrangement  of  systems  that  results  when  independent  and  useful  systems  are  integrated  into  a 
larger  system  that  delivers  unique  capabilities.  DoD  Defense  Acquisition  Guide,  System  of  Systems 
Engineering 


Accomplishments  and  Plans 


•  Completed  SoS  SE  Guide  v.9  in  December  2006 


•  Executed  six  month  pilot  phase 

-  Identified  key  SoS  SE  elements  and  principles 

-  Identified  SoS  SE  issues  which  require  further  attention 


•  Socializing  insights  (SE  Forum,  incose,  nasa,  sstc 
Conference,  NDIA,  others) 


•  Next  Steps 

-  Update  SoS  SE  Guide  with  pilot  findings 

-  Update  DoD  SE  Guides  (SEP,  DAG)  for  SoS  considerations 

-  Plan  for  DAU  Continuous  Learning  Module  in  FY08 

-  I  implement  FY08  activities  to  address  identified  issues 


A  mechanism  to  share  emerging  insights  on  SoS  and  implications  for  SE 
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Pilot  Participants 


Objective  of  the  pilots 
was  to  gain  a 
'boots  on  the  ground' 
perspective 


Research  Community 

I NCOSE:  International  Council  on  SE 
MIT:  Massachusetts  I  nstitute  of  Technology 
MITRE:  MITRE  Corporation 
Purdue:  School  of  Engineering 
SEI :  Software  Engineering  Institute 
Stevens:  Institute  of  Technology 
USC:  University  of  Southern  California 
UCSD:  University  of  California  San  Diego 
Australia:  Defence  Materiel  Organisation 


SE  Practitioners 

ABCS:  Army  Battle  Command  System 

AOC:  Air  Operations  Center 

BMDS:  Ballistic  Missile  Defense  System 

CAC2S:  Common  Aviation  Command  &  Control  System  I 

DCGS-AF:  Distributed  Common  Ground  Station  (mitre) 

DoDII  S:  DoD  I  ntelligence  I  nformation  System  (mitre) 

FCS:  Future  Combat  Systems 

MILSATCOM:  Military  Satellite  Communications 

Nl  FC-CA:  Naval  I  ntegrated  Fire  Control  -  Counter  Air 

SR:  Space  Radar 

NSA:  National  Security  Agency 

NSWC:  Naval  Surface  Warfare  Center  Dahlgren 

PEO  GCS:  Ground  Combat  Systems 

SI  AP:  Single  I  ntegrated  Air  Picture 

SMC:  Space  and  Missile  Systems  Center 

TMI  P:  Theater  Medical  I  nformation  Systems  -  J  oint 

USGC:  US  Coast  Guard  C2  Convergence  (MITRE) 

o 
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Emerging  Insights  from  SoS  Pilots 

SoS:  I  s  1 1  New? 


'ns 


•9hts 


.SSS 


•  Most  military  systems  today  are  part  of  an  SoS  whether  or  not 
explicitly  recognized 

-  Most  systems  are  created  and  evolve  without  explicit  SE  at  the  SoS 
level 

•  A  formal  SoS  comes  into  existence  when  something  occurs  to 
trigger  recognition  of  SoS 

•  An  organization  is  identified  as  'responsible  for'  the  SoS  'area' 
along  with  definition  of  the  objective  of  the  SoS 

-  Does  not  include  changes  in  ownership  of  the  systems  in  the  SoS 

•  The  SoS  is  then  structured 

-  Membership  is  defined  starting  with  identification  of  systems  in  the  SoS 

-  Processes  and  organizations  are  established  for  the  SoS,  including  SE 


SoS  in  the  DoD  is  not  new; 

Recognizing  SoS  in  development,  and  recognizing  SoS  SE  is  new 
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What  Does  SoS  Look  Like  in 
the  DoD  T oday? 


'ns 


•9hts 


•  Typically  an  overlay  or  ensemble  of  individual  systems 
brought  together  to  satisfy  user  capability  needs 

•  Not  new  acquisitions  per  se 

-  Cases  like  FCS  are  extremely  rare  and,  in  practice,  still  must 
integrate  with  legacy  systems 

•  SoS  'manager'  does  not  control  the  requirements  or 
funding  for  the  individual  systems 

-  May  be  in  a  role  of  influencing  rather  than  directing,  impacts 
SE  approach 

•  Focus  of  SoS  is  on  evolution  of  capability  over  time 

•  A  functioning  SoS  takes  start-up  time  but,  in  steady 
state,  seems  well-suited  to  routine  incremental  updates 


Most  military  systems  are  part  of  an  SoS  operationally 
Only  by  exception  do  we  manage  and  engineer  at  SoS  level 
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•  Translating  SoS  capability  objectives  into  high  level 
requirements  over  time 


•  Understanding  the  systems  in  the  SoS  and  their 
relationships 

•  Assessing  extent  to  which  the  SoS  meets  capability 
objectives  over  time 

•  Developing,  evolving  and  maintaining  a  design  for  the 
SoS 

•  Anticipating  and  assessing  impacts  of  potential  changes 
on  SoS  performance 

•  Evaluating  new  and  evolving  requirements  on  SoS  and 
options  for  addressing  these 

•  Orchestrating  upgrades  to  SoS 


The  SoS  SE  is  responsible  for  creation  and  continual  application 
of  approaches  to  accomplish  these  elements 
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External  Influences 


Relationships  Among 
SoS  SE  Elements 
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Translating  capability  objectives 


Understanding  systems  &  relationships 


1  1 1 


i 


► 

► 


Assessing  performance  to  capability  objectives 


n 


TT 


Developing,  evolving  and  maintaining  SoS  design 


Mtll 


Monitoring  &  assessing  changes 


I 


Addressing  new  requirements  &  options 


Orchestrating  upgrades  to  SoS 


T 
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Typically  not  the 
role  of  the  SE  but 
key  to  SoS 


Block  upgrade 
process  for  SoS 


Large  role  of 
external  influences 
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Relationship  Among  Core  Elements  of  SoS  SE 


Translating 

capability 

objectives 


nderstanding 
systems  &  ' 
relationships 
■includes  plans 


Orchestrating 
upgrades 
to  SoS 


Addressing 

new 

requirements 
&  options 


Assessing 
performance 
to  capability 
objectives 


Developing, 
evolving  and 
maintaining 
SoS  design 


Monitoring 
&  assessing 
changes 


10/26/200/ 


External  Environment 
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What  is  Working? 
SoS  SE  Principles 


•  Address  organizational  as  well  as  technical  perspectives 

•  Focus  on  areas  critical  to  the  SoS 

-  Leave  the  rest  (as  much  as  possible)  to  the  SEs  of  the  systems 

•  Technical  management  approach  reflects  need  for 
transparency  and  trust  with  focused  active  participation 

•  SoS  designs  are  best  when  open  and  loosely  coupled 

-  I  mpinge  on  the  existing  systems  as  little  as  possible 

-  Are  extensible,  flexible,  and  persistent  overtime 

•  Continuous  ('up  front')  analysis  which  anticipates  change 

-  Design  strategy  and  trades  performed  upfront  and  throughout 

-  Based  on  robust  understanding  of  internal  and  external  sources  of 


change 
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Relationship  to 
Core  SE  Processes 


•  16  SE  processes  apply  across  the  SoS  SE  elements 

•  Offer  a  toolbox'  to  apply  to  SoS  SE  needs 


Technical  Processes_ Technical  Management  Processes 


SoS  SE 
Elements 

Rqts 

Devel 

Logical 

Analysis 

Design 

Solution 

Implement 

1  ntegrate 

Verify 

Validate 

Transition 

Decision 

Analysis 

Tech 

Planning 

Tech 

Assess 

Rqts  Mgt 

Risk  Mgt 

Gonfig 

Mgt 

Data  Mgt 

Interface 

Mgt 

T ransl  ati  ng  Capabi  1  ity 
Objectives 

X 

X 

X 

Understanding  Systems  and 
Their  Relationships 

X 

X 

X 

X 

X 

X 

Assessing  Performance  to 
Capabi  1  ity  Obj  ectives 

X 

X 

X 

X 

X 

X 

Developing,  Evolving  & 
Maintaining  SoS  Design 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Monitoring  and  Assessing 
Changes 

X 

X 

X 

Address  New  Rqts  & 
Options  to  1  implement 

X 

X 

X 

X 

X 

X 

X 

X 

Orchestrating  Upgrades 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Reflect  the  fact  that  technical  Reflect  the  SoS  SE  role  of 

processes  are  primarily  technical  coordination  and 

implemented  by  systems  direction  across  systems 
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External  Influences 


I  information  Flow 
Among  SoS  SE  Elements 


Inputs: 

Stakeholder  needs 
Threat  conditions 
National  priorities 


I 


I 


Translating 

capability 

objectives 


Outputs: 

First  order 
SoS  goal  and 
expectations 


I 


Monitoring 
&  assessing 
changes 


inputs: 

User  needs  based 
on  operational 
feedback 
Outputs: 

First  order  SoS  goal 
and  expectations 


Inputs: 

Design  feasibility 
Outputs: 

First  order  SoS 
goal  and 
expectations 


Assessing 
(actual) 
performance 
to  capability 
objectives 


Inputs: 

Status  of 
systems  and 
functionality 
Outputs: 

First  order  SoS 
goal  and 
expectations 


Developing, 
evolving  and 
maintaining 
SoS  design/  arch 


Understanding 
systems  &  “ 
relationships 
(includes  plans) 
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SE  Processes  Supporting 
Each  SoS  SE  Element 


Translating  Capability  Objectives  (sample) 
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'The  Requirements 
Development  process 
takes  all  inputs  from 
relevant  stakeholders  and 
translates  the  inputs  into 
technical  requirements." 
[DAG] 

•  Top  level  capability  objectives  ground  the 

requirements  for  the  SoS 

•  1  n  an  SoS,  in  most  cases  requirements 
development  is  an  ongoing  process. 

•  As  the  SoS  evolves  over  time,  needs  may 
change.  The  overall  mission  may  be  stable,  but 
the  threat  environment  may  be  very  different. 

•  1  n  a  SoS,  capability  objectives  may  be  more 
broadly  conceived  . . . 

•  .  .  . 

"Requi  rements 
Management  provides 
traceability  back  to  user- 
defined  capabilities... 
"[DAG] 

•  The  requirement  management  process  begins  with 
translating  SoS  capability  objectives  into  high  level 
requirements  in  the  SOS  SE  process.  The  work  in 
this  element  provides  the  grounding  for  the  work 
done  over  time  in  defining,  assessing,  and 
prioritizing  user  needs  for  SoS  capabilities. 

•  . 

"Data  management . . . 

addresses  the  handling 
of  information  necessary 
for  or  associated  with 
product  development 
and  sustainment."  [DAG] 

•  Translating  SoS  capability  objectives  into  high 
level  requirements  is  the  start  point  of  building  a 
knowledge  base  to  support  the  SoS  development 
and  evolution. 

•  1  n  this  element  the  SE  develops  and  retains  data 
on  the  the  capability  needs  and  high  level 
requirements  for  the  SoS  for  use  throughout  the 

SoS  elements. 
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Comparison  of 

Engineering  Focus  Areas  a  of  2) 


Area 

Systems 

System  of  Systems 

What  to 
engineer 

Based  on  a  set  of 
functional  and 
performance 
requirements  for 
the  system  of 
interest 

•  Based  on  a  set  of  SoS  capabilities  that  are 
then  translated  into  high  level  requirements 
for  further  analysis 

•  A  single  capability  can  result  in  multiple 
requirements  that  affect  multiple  constituent 
systems 

View  of 
system- 
of- 

interest 

Clear  system 
boundaries 

1  nterfaces 

•  Systems  that  contribute  to  SoS  capabilities 
and  the  interrelationships  between  those 
systems 

Architect 

ure 

Developed  and 
optimized  to 
support  single 
purpose  of  system 

•  Net- centric,  focused  on  information  sharing 

•  Does  not  address  design  details  within 
constituent  systems,  but  rather  the  way  the 
systems  work  together  to  meet  user  needs 

•  Sufficient  versus  optimized 

Design 

approach 

10/26/2007 

Often  top-down 

•  Combined  top-down  and  bottom- up,  with 
focus  on 

-  Existing  assets  (systems)  that  are  within 
the  SoS 

-  Opportunities  within  constituent  syst^n 
lifecycles  for  changes 

Comparison  of 

Engineering  Focus  Areas  (2  of  2) 


Area 

Systems 

System  of  Systems 

1  mplementation 

Contract- 

controlled, 

often  using  an 

incremental, 

evolutionary, 

or  spiral 

process 

Focus  on  total 
system 

•  SoS  functionality  implementation 
accomplished  through  combination  of 
negotiation,  sometimes  funded  by  SoS 
or  system  owner,  not  always  done  via 
formal  agreements 

•  Asynchronous  and  incremental  due  to 
lifecycles  of  constituent  systems 

•  Primarily  concerned  with  the 
implementation  of  SoS  functionality, 

•  Monitors  the  evolution  of  constituent 
systems  to  ensure  that  SoS  is  not 
adversely  impacted,  but  not  typically 
involved  in  the  implementation  details 

Testing 

10/26/2007 

Traditional 

testing 

activities,  e.g., 
DT&E  and 

OT&E 

•  Attempt  to  leverage  off  of  constituent 
system  testing 

•  Often  impossible  to  test  full-up  SoS  in  a 
lab— often  rely  on  constituent  system 
integration  labs  and  operational  testing 

•  Operationally,  looking  for  how  users 
use  the  system  and  identifying 
emergent  behavior  for  further  analysis 

I  ssues  to  be  Addressed 


•  Testing  in  a  systems  of  systems  environment 

•  SoS  risk  and  cost  drivers  - 


-  I  dentify  and  plan  for;  mitigate  interdependency  risk 

-  I  nform  leadership  of  risk 

•  Community  questions  - 


Ongoing  SoS 
!PT  Exchange 


Briefed  to 
T&EDSB 


FY08SSE 

Initiative 


-  Should  we  change  the  way  we  engineer  individual  systems? 

-  What  is  the  role  of  net-centricity  in  SoS? 

•  Enablers  to  allow  SEs  to  better  operate  in  SoS 
environments,  such  as 


iNCOSE 

Working 

Group 


-  Additional  processes  or  new  ways  to  implement  current  processes 

-  New  contracting  methods 

-  New  models  of  governance 
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Summary  and  Discussion 


•  US  plans  to  continue  SoS  project  in  FY08  and 
beyond 

-  Publish  SoS  Guide  Version  1.0 

-  Update  SE  policy/guidance/training  with  SoS  findings 

-  Address  open  issues 

-  Apply  findings  to  program  support  activities 

-  Apply  findings  to  portfolio  managers  -  C2,  J  NO, 
others 
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Backup  Slides 
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Definitions 


System 

An  integrated  composite  of  people,  products,  and  processes  that 
provide  a  capability  to  satisfy  a  stated  need  or  objective 

Mil-Std  499B 

System  of  Systems 

A  set  or  arrangement  of  systems  that  results  when  independent 
and  useful  systems  are  integrated  into  a  larger  system  that 
delivers  unique  capabilities 

DoD  Defense  Acquisition  Guide,  System  of  Systems  Engineering 

System  of  Systems  Engineering 

Planning,  analyzing,  organizing,  and  integrating  the  capabilities  of 
a  mix  of  existing  and  new  systems  into  a  SoS  capability  greater 
than  the  sum  of  the  capabilities  of  the  constituent  parts 

DoD  Defense  Acquisition  Guide,  Chapter  4 
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Acquiring  Defense  Capabilities 

SoS  SE  Considerations 


•  Ownership/  Management  I  ndividual  systems  are  owned  by  the 
military  Services  or  agencies 

•  Legacy  Current  systems  will  be  part  of  the  defense  inventory  for  the 
long-term  and  need  to  be  factored  into  any  approach  to  SoS 

•  Changing  Operations  Changing  threats  and  concepts  mean  that 
new  (ad  hoc)  SoS  configurations  will  be  needed  to  address  changing, 
unpredictable  operational  demands 

•  Criticality  of  Software  SoS  are  constructed  through  cooperative  or 
distributed  software  across  systems 

•  Enterprise  I  ntegration  SoS  must  integrate  with  other  related 
capabilities  and  enterprise  architectures 

•  Portfolios  SE  will  provide  the  technical  base  for  selecting 
components  of  the  systems  needed  to  support  portfolio  objectives 


id 


Capability  needs  will  be  satisfied  by  groupings  of  legacy 
systems,  new  programs,  and  technology  insertion  - 

Systems  of  Systems  (SoS) 


9 


System  of  Systems  - 
The  Management  Challenge 


SoS: 

Within 
Single 

Organization 


IV  A.  .14-: . —  I  — 

Political  and  Cost  Considerations  I  mpact  on  Technical  Issues 

v_/i  yai  iiz-ciLKJi  iz> 
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I  nitial  Pilot  Results 


Wide  range  of  views  on  the  SoS  depictions 

-  Still  sorting  out  a  good  approach,  inputs  welcome 

-  Most  felt  current  depictions  did  not  adequately  portray  the 
dynamics  and  complexity  faced  in  SoS  SE 

General  agreement  on  Systems  vs  SoS  distinctions 

-  Need  for  more  careful  wording 

-  Particular  need  to  clarify  discussion  of  'stakeholders' 

Most  felt  that  the  guide  needed  an  explicit 

discussion  of  SoS  and  SoS  SE  in  the  DoD  today 

-  Need  to  describe  the  elements  of  SoS  SE  and  clearly 
differentiate  between  the  role  of  the  SoS  SE  and  the  System 
SEs  in  SoS 

-  Provide  context  for  discussion  of  16  processes 

16  SE  processes 

-  General  agreement  that  these  apply  to  SoS  and  with  the 
thrust  of  the  discussion  on  each  process 

-  Need  to  clarify  how  these  are  implemented  at  the  SoS  and 
how  these  relate  to  the  same  processes  for  the  systems 

Guide  too  long  and  hard  to  use 
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Introduction 


•  SoS  Systems  Engineering  project  addressing  LCS 
ASW  Integration  &  Mission  Capability  Evolution 

•  Pilot  project  conducted  by  ASW  Systems  Engineering 
Team  (ASSET)  chaired  by  PEO-IWS5  SE 

•  Application  of  ASN/RDA  CHSENG  Naval  SoS  SE 
Guidelines 

•  Employed  Quality  Function  Deployment  (QFD)  for  SoS 
capability  evaluation 


10th  Annual  Systems  Engineering  Conference 
Session  -  T&E  in  Systems  Engineering 


ASW  SoS  Systems  Engineering  Pilot  - 
QFD  Analysis 


LCS  ASW  SoS  Pilot  Project 


•  Proliferation  of  quiet  diesel  submarines 
creates  a  growing  ASW  challenge 

•  ASW  inherently  a  “system-of-systems” 
enterprise: 

•  Platforms 

•  Sensors 

•  Weapons 

•  Command,  Control  &  Communications 

•  Littoral  Combat  Ship  (LCS)  a 
“transformational”  concept: 

•  Agile  platform 

•  Reconfigurable  mission  packages 

•  Extensive  use  of  unmanned  vehicles  & 
off-board  sensors 

•  Spiral  development 

•  Pilot  project  objectives 

•  Address  needed  ASW  capability 

•  Apply  ASN/RDA  SoS  SE  guidelines 

-  Including  QFD 

•  Show  “value  added”  in  SoS  acquisition 


LCS  Platform  Concepts 
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ASW  System  of  Systems  Engineering  Process 


ASW 

Capability 

1 


JCIDS 


ASW  (FoS) 
Capability 
Planning 


•AOAs 
•CDDs 
•CPDs 
•NR-KPP 


T 


SoS 

Acquisition 

Portfolio 

Execution 


ASW 

Capability 

1 

Acquisition 

Programs 


Acquisition"!— 


ASW 

ipability 
2 

iquisition 

’rograms 


System  Performance 

Document  (SPD) 

•SoS  Performance 
Requirements 
•SoS  Functional 
Architecture 
•SoS  Physical 
Architecture 
•SoS  T&E 
Requirements^ 


L4SW 
p  ability 
n 

juisition 
rograms  J 


? - - 

SoS  Capability 
Engineering 


Pilot 

Assessment 


Synchronized 
Delivery  of 
ASW 

Capabilities 


•ASW  Capabilities  & 
Objectives 
•ASW  Architecture 
Development/Mgt 
•ASW  Portfolio  ID 
•ASW  Portfolio  Roadmlar 


SoS  Capability 
Planning 


Capability  Evolution  Plan  (CEP) 

•Pilot  CONOPS 

•Force  Package  Structure 

•Quality  Function  Deployment  (QFD) 

•System  Acquisition  Roadmap 

•SoS  Capability  Assessment _ 


Connotes  SoS  SE  Steps  4 
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Session  -  T&E  in  Systems  Engineering  QFD  Analysis 


LCS  ASW  Mission  Context 
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10th  Annual  Systems  Engineering  Conference 
Session  -  T&E  in  Systems  Engineering 


ASW  SoS  Systems  Engineering  Pilot 
QFD  Analysis 


Force  Package 


LCS  Pilot  Project  Portfolio 

Offensive  ASW  Capability 
(Barrier  Mission) 


Sub 


Air 


Surf 


Surv  (Fixed  &  Mobile) 


Other  Tactical 
Assets 


Spiral  A  ASW 
Mission  Package 


ASW  Modules 
& 

Systems  (Defined) 


Collaborative 

Operations 


LCS 

(Platform  A/B) 


Spiral  B  ASW 
Mission  Package 


ASW  Modules 
& 

Systems  (TBD) 


Other  Activities 


1 

C4ISR/Net 


CUP/COP 


Mission  Plan. 


Remote  Processing 


Logistics 
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ASW  SoS  Systems  Engineering  Pilot 
QFD  Analysis 


QFD  Matrices  for  Capability-Based  Planning 


10th  Annual  Systems  Engineering  Conference 
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ASW  SoS  Systems  Engineering  Pilot 
QFD  Analysis 


Pilot  QFD  Matrices  &  Workshop 


Mission  & 
SoS  Systems 
MCA 


Matrix  1 


Operational  & 
Engineering  Metrics 
(ICD,  CDD,  Other) 


Matrix  2 
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Importance  Score 
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Functions 
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c 

o 

§  I 
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c n 
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o 
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SoS  Capability  Metrics 


Functional  Capability 
Assessment 


Capability  Score 


Workshop  Day  2 


2-Day  Workshop 

~30  Subject  Matter  Experts  (SME) 

Divided  into  four  teams 

Operational,  technical,  engineering  expertise 


SoS  Capability 
“Gaps” 
(CEP  Focus) 
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Matrix  1  (partial) 

Operational  Activity  &  System  Functions 


Matrix 

Rows  &  Columns 
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Importance  score  =  2  (Activity  value  X  criticality  value  per  cell) 


10th  Annual  Systems  Engineering  Conference 
Session  -  T&E  in  Systems  Engineering 


ASW  SoS  Systems  Engineering  Pilot 
QFD  Analysis 


Stage  Score  Allocations  (Matrix  1 ) 


Matrix  1  (Day  1) 


(0 

a> 

o  CO 

SoS  Functions  &  Systems 

(0  (1) 

55  5 
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o  < 

Functional  &  System 

'll 

Importance 

Q. 

o 

Importance  Score 

450 
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350 
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Team  Average 


Team  Scores 


200 


100 


Storage 


Deployment 


Installation 


Execution 


Self-Defense 


Mission  Sustain.  & 
Post-Mission  OPS 


Stages 


Team  prioritization  of  operational  the  six  stages 
(Allocation  of  1000  points) 
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QFD  Analysis 


Operational  Activity  Priority  -  Execution  Stage 
Rank  Ordered  Team  Averages 

(Allocation  of  points  assigned  to  Execution  Stage) 
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O 

—  a> 
co  « 

«> 

“-o 

°<C 
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D.2 

D.5 

D.14 

D.3 
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D.1 
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D.13 
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Matrix  1  (Day  1 ) 
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Importance  Score 
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Function  Importance  (Execution  Stage/Matrix  1) 

Team  Score  Averages  (Ranked) 


Matrix  1  (Day  1) 


(0 
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Q. 

o 

Importance  Score 

System  Functions  (ID) 
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System  Importance  (Matrix  1 ) 
Execution  Stage 
Ranked  Team  Averages 
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Matrix  2  (partial) 

System/Functions  vs  Capability 


Matrix  2  (Day  2) 


SoS  Capability  Metrics 


Functional  Capability 
Scoring 


Capability  Score 


Step  1 :  Address  higher 
important  functions  and 
systems  from  Matrix  1 


Capability  Metrics  (by  Category) 


System  and  Functions  (from  Matrix  1) 


M1.1  ASW  Search 


Search 

Rate/Coverage 


False  Contact 
Rate 


LCS  Mission 
Sustainment 
(days) 


Area  Environmental 
Understanding 


Ml  .2  Kill  Chain  Metrics 


Redetect  & 
Class 


Asset 

Availability/Respo 

nsiveness 


Weapon 

Availability 


Functional 

Component 


Platform  /  Vehicle 


System 


Metric  Score  (frorr 
Matrix  #1) 


1.1. 1.1  Helo  (SH-60B, 
MH-60R) 


FI. 1.1. 1.2 
FI. 1.1. 1.3 


Communicate _ 

Sensor  Processing 


1.1.1 .2  ALFS  (Dipping 
Sonar)  (MH60R  Only) 


1.1  Aviation 
Mission  Module 


1.1.1  Helicopter 


1.1. 1.3  Sonobuoy 
(Family) 


FI. 1.1. 3.1 
FI. 1.1. 3.2 


Onboard  Processing 


FI. 1.1. 3.3 
FI. 1.1. 3.4 


Localize _ 

Onboard  Processing 


Communicate 


1.1 .1.4  MAD,  Radar, 
EO  (Helo  Dependent) 


Detect/Classify/Localize 


1.1. 1.5  Weapons 
(Mk46,  MK54) 


FI. 1. 1.5.1 
FI. 1.1. 5.2 


Deploy  /  Placement 
Search 


FI. 1.1. 5.3 
FI. 1.1. 5.4 


1.2  Mid- 
Frequency 
Bistatic  Mission 
module 


1. 2.1.1  RTA/1.2.1.2 
RTAS 


FI  .2. 1/2.1 
FI .2. 1/2.2 


Operate  &  Control 


Localize  /Track 


Communicate _ 

Onboard  Processing 


1.3  LF  Bistatic 
Mission  Module 


1. 3.1.1  UTAS  / 1.3. 1.2 
MSOBS 


FI .3.1. 1/2.2 
FI  .3.1. 1/2.3 
FI  .3.1. 1/2.4 


Operate  &  Control 


Sense 
Localize /Track 
Communicate 


Status  /  Diagnostics 


1.4  Mid- 
Frequency 
Monostatic 
Mission  Module 


1.3.1/ 1.4.1  USV(2) 
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o 

o 
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m 

Q. 
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O 

75 

a_ 
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FI  .4.1. 
FI  .4.1. 
FI  .4.1. 


System  Capability  Score 


Step  2:  Score  in  terms  of 
capability  (fully  capable  to 
significantly  incapable) 


Step  3:  Compile  overall  capability  scores 

Score  =  I  (system  function  score  X  adequacy  rating  value  per  cell) 


- ► 

Additional 

Matrix 

Rows  &  Columns 
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Capability  Scores 

(Execution  Stage) 


Matrix  2  (Day  2) 


C 

O 

SoS  Capability  Metrics 

i  i 

LL  fij 

E  si 

Functional  Capability 

£  <o 

CO 

Scoring 

CO 

o 

_ CO _ 

Capability  Score 

Moderately  _ 

Capable 


Moderately 

Inadequate 

Completely 

Inadequate 


Search 


Enqaqement 


Redetect  &  Class  Localize  Kill  Engage  Weapon  Availability  Asset 

Availability/Res  ponsiwness 

Capabilities 


Moderately 

Capable 


Moderately 

Inadequate 


Completely 

Inadequate 


Score  <  6  considered  a  Capability  “gap” 
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System  Capability  vs.  Importance 
(Results  of  Matrix  1  &  2) 
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System  Importance  Score  (Matrix  1) 


(Based  on  Execution  Stage  evaluation) 
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QFD  Workshop  Comments 

•  Carefully  constructed  matrices  critical  to  success 

•  Manageable  matrix  size  (dimensions) 

•  A  two-day  workshop  was  insufficient 

•  Dividing  participants  into  four  smaller  working  groups  worked  well. 

•  Need  clear  Concept  of  Operations  and  mission  threads  (ideally  an 
approved  set  of  architectures) 

•  Description  and  performance  information  regarding  the  systems  being 
rated  needed  on  site 

•  An  experienced  QFD  workshop  facilitator  if  not  facility  recommended 
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Summary 


•  LCS  ASW  Integration  Pilot  project  has  been  a  good  example  of  SoS  SE 
process 

-  Portfolio  of  systems 

-  Application  of  the  ASN/RDA  SoS  SE  Guidelines 

•  The  QFD  process  was  adapted  from  the  SoS  SE  Guide  and  other  QFD 
applications  and  was  effective  in  identifying  functional  priorities  and 
capability  gaps  across  a  complex  SoS  portfolio. 

•  QFD  matrices  must  be  customized  to  assess  the  operational,  functional, 
and  physical  aspects  of  the  Force  Package. 

•  The  matrices  map  to  or  expand  upon  the  DOD  Architecture  Framework  and 
thereby  are  a  further  use  of  the  architecture  products 

•  The  process  followed  is  considered  useful,  applicable,  and  adaptable  to 
other  SoS  capability  evolution  scenarios. 
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ASW  SoS  Systems  Engineering  Pilot  - 
QFD  Analysis 


Backups 
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ASW  SoS  Systems  Engineering  Pilot  - 
QFD  Analysis 


LCS  Operational  Stages  and  Activities  (Matrix  1 ) 

Activity  Activity  Activity 

Module  configuration  and 
checkout 

Transport _ 

Onload  modules  and 
mission  crew 


Plan  ASW  mission _ 

Underway  and  Transit 
Arrive  at  assigned 
operating  area _ 

Establish  theater  tactical 
communications  (OPCON 

and  TACON) _ 

Coordinate  with 
other  assets 


Characterize/measure 
area  environment 
Mission/Sensor 
employment  planning 
Final  vehicle/sensor 
reconfiguration  and 
checkout _ 

Launch 

vehicles/OOV's _ 

Check  operability 
Return  to  patrol  station 
Operate  and  monitor 
OOV's 


Conduct  area  search 
Detect  and  classify 
contacts 

Resolve  possible  false 
contacts 

Report  detections  to  Sqn 

and  ASWC _ 

Localize,  track  and 
monitor  threat 
submarines 
Target  reported  to  sqn 
and  ASWC 
Prosecution  assets 
proceed  to  target 
Prosecution  assets 
redetects,  classifies  and 
localizes  target 
Prosecution  assets 
request  and  receives 
attack  authorization 
Prosection  assets 
launches  weapon 
Attack  assessment 
Reattack  if  required 
Prosecution  assets 
return  to  LCs  or  patrol 
station 

Handoff/  receive 
targets  with  other 
assets 

Maintain  tactical  Picture 


0) 

■  c /> 

^  c 

Threat  /Weapons  DCL 

<D  0) 

( 0 

a 

Evade 

Refurbish  and  Redeploy 

OOV's _ 

LCS  proceeds  to  sensor 
station _ 

Final  recovery  of  OOV's 
Conducts  turnover  with 

relieving  LCS _ 

Onboard  stowage _ 

Transit  to  port  (or  ship 
replenishment  sight) 

Off-load _ 

Refurbishment _ 

Stowage 


Ref:  LCS  ASW  Mission  Package  Overview, 
PMS  420 


**Ref:  LCS  Operational  Assessment 
Scenario,  SPA  and  other  sources 
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LCS  ASW  SoS  Pilot 
System  and  Functions  (  Matrix  1  &  2) 


1.  OOV/Sensors 


1.1.1  Helicopter 

1.1 .1.1  Helo  (SH-60B,  MH-60R) 

1.1. 1.2  ALFS  (Dipping  Sonar)  (MH60R 
Only) 

1.1. 1.3  Sonobuoy  (Family) 

1.1. 1.4  MAD, 
Radar,  EO 
(Helo 

Dependent) 

1.1 .1.5  Weapons  (Mk46,  MK54) 

FI  .1.1.1 .1 

FI. 1. 1.1.2 

FI. 1. 1.1.3 

FI. 1.1. 1.4 

FI. 1.1 .2.1 

FI. 1.1 .2.2 

FI .1.1. 2.3 

FI .1.1. 2.4 

FI  .1.1. 3.1 

FI  .1.1. 3.2 

FI. 1.1. 3.3 

FI. 1.1. 3.4 

FI. 1.1. 3.5 

FI  .1.1 .4.1 

FI. 1.1 .5.1 

FI  .1.1 .5.2 

FI  .1.1. 5.3 

FI  .1.1. 5.4 

Flight 

operations 

Communicate 

Sensor 

Processing 

Navigation 

Deploy 

Sense 

Localize 

Onboard 

Processing 

Deploy 

Sense 

Localize 

Onboard 

Processing 

Communicate 

DCL 

Deploy  / 
Placement 

Search 

Acquire 

Target 

Kill 

1.2  Mid-Frequency  Bistatic  Mission  module 

1.3  LF  Bistatic  Mission  Module 

1.4  Mid-Frequency  Monostatic  Mission  Module 

1.5  UAV  Mission  Module 

1.6  Mission  Package  Support 

1.2.1  RMV(2) 

1.3.1/ 1.4.1  USV(2) 

1.5.1  VTUAV 

1.6.1  MPSE/COMMS/Storage 

1. 2.1.1  RTA/1.2.1.2RTAS 

1. 3.1.1  UTAS  / 1.3. 1.2  MSOBS 

1.4.1.  UDS 

1. 5.1.1  VTUAV  Payload 

1. 6.1.1  MPCE 

1.6.1. 2 
OOV 
COMMS 

1.6.1 .3 
Storage 

FI. 2.1/2. 1 

F1.2.1/2.2 

F1.2.1/2.3 

F1.2.1/2.4 

F1.2.1/2.5 

F1.2.1/2.6 

F1.3.1 .1/2.1 

FI  .3.1 .1/2.2 

FI  .3.1. 1/2.3 

F1.3.1.1/2.4 

FI  -3.1 .1/2.5 

FI  .3.1. 1/2.6 

FI -4.1.1 

FI  .4.1 .2 

FI  .4.1 .3 

FI -4.1 .4 

FI  .4.1 .5 

FI -4.1. 6 

FI  .5.1 .1.1 

FI  .5.1 .1.2 

FI  .5.1 .1.3 

FI  .5.1 .1.4 

FI  .5.1 .1.5 

FI  .5.1 .1.6 

FI  .5.1. 1.7 

FI -6.1.1 

FI  .6.1 .2 

FI  .6.1 .3 

FI  .6.1 .4 

FI  .6.1 .5 

FI  .6. 1.2.1 

FI  .6.1 .3.1 

Operate  & 
Control 

Sense 

Localize  / 

Communicate 

Onboard 

Processing 

Navigate 

Operate  & 
Control 

Sense 

Track 

Communicate 

Diagnostics 

Navigate 

Control 

Sense 

Localize  / 

Communicate 

Navigate 

Onboard 

Processing 

Operate  & 
Control 

Communicate/ 

Relay 

Sense 

Classify 

Localize 

Attack? 

BDA 

Data  Fusion  & 
Contact 
Management 

CAUSS 

Display 

Mission 

Planning 

Operations 

Control  & 
Data  Links 

Weapons, 

HAZMAT, 

CPG 

2.  Host  Platfo 

rm 

2.1.1 .1 
Crew 

2.1 .1.2  Communications  (Various  Radios 
&  Nets)) 

2.1. 1.3  MS  Handling 
Systems 

2.1. 1.4  Mission  Package  Support  Equipment 

2.1.1 .5  Command  &  Control 

2.1. 1.6  Ship  Defense 

F2.1. 1.1.1 

F2.1. 1.2.1 

F2.1. 1.2.2 

F2.1. 1.2.3 

F2.1. 1.2.4 

F2.1. 1.3.1 

F2.1. 1.3.2 

F2.1. 1.4.1 

F2.1. 1.4.2 

F2.1 .1.4.3 

F2.1. 1.4.4 

F2.1. 1.4.5 

F2.1 .1.5.1 

F2.1. 1.5.2 

F2.1. 1.5.3 

F2.1. 1.5.4 

F2.1. 1.6.1 

F2.1. 1.6.2 

F2.1. 1.6.3 

F2.1. 1.6.4 

Ship 

Operations 

Ship-Ship 

Ship-to-Shore 

Ship  to  Off- 
Board 
Systems 

Ship  to  Force 
ASW  Assets 

MM  Deploy 
Crew 

Deploy/Recover 

MM 

MP  Control 

Data 

Processing 

Display 

Mission 

Planning 

Test  MP 

Mission 

Planning 

Env.  Data 
Gathering 

Coordination 

Common 

Processing 

Weapon  DCL 

Counter 

Evade 

Mission 

recovery 

4.  Theater  Assets 

4.1.1  CSG/ESG  Platforms 

4.1.2  Other  Assets 

4.1. 1.1 
Network 
(GIG/Forc 
eNet) 

4.1. 1.2  ASW 
Command  & 
Control 

4.1. 1.3  Common 
Picture 

4.1. 1.4  Mission  Planning 

4.1. 2.1  P8 
A,  P3-C 

4.1. 2.2 
Force 
Helo's 

4.1. 2.3 
Other 
ASW 
Assets 

4.1. 2.4 

Theater 

ISR 

F41.1.1.1 

F4.1 .1.2.1 

F4.1. 1.2.2 

F4.1.1.3.1 

F4.1. 1.3.2 

F4.1. 1.4.1 

F4.1. 1.4.2 

F4.1. 1.4.3 

F4.1.2.1.1 

F4.1 .2.2.1 

F4.1. 2.3.1 

F4.1. 2.4.1 

Communicati 

ons 

ASWC 

TASW 

Common 

T  actical 
Picture 

Common 

Operational 

Picture 

Area 

assignment 

Sensor 

employment 

Mutual 

Interfeerence 

Cooperative 

ASW 

Cooperative 

ASW 

Cooperative 

ASW 

Cueing 

3.1  Maintenance  and  Storage 

3.1.1  MP  Shore/IMA/Depot 

3. 1.1.1  MP  Shore/IMA 

3.1. 1.2 
Depot  & 
OEM 

F3. 1.1. 1.1 

F3.1.1.1.2 

F3.1.1.1.3 

F3.1 .1.2.1 

Train  MP 
personnel 

Store  & 
Maintain  Equip 

Transport 

MP 

Equipment 

Accept  & 
Refurb  Equip 
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Capability  Performance  Metrics 

(Matrix  #2) 


M1.1  ASW  Search 

Ml  .2  Kill  Chain  Metrics 

Ml. 1.1 

Ml. 1.2 

Ml. 1.3 

Ml. 1.4 

Ml. 1.5 

Ml. 1.6 

Ml. 2.1 

Ml. 2.2 

Ml. 2.3 

Ml. 2.4 

Ml. 2.5 

Ml. 2.6 

Search 

Rate/Coverage 

Detect/Class 

Track 

False  Contact 
Rate 

LCS  Mission 
Sustainment 
(days) 

Area  Environmental 
Understanding 

Redetect  & 
Class 

Localize 

Engage 

Kill 

Asset 

Availability/Respo 

nsiveness 

Weapon 

Availability 

M2  System  Employment  Metrics 

M3  SOS  Metrics 

M2.1 

M2. 2 

M2. 3 

M2.4 

M2.5 

M3.1 

M3. 2 

M3.3 

M3.4 

Deploy/Recovery 

Feasibility 

Operating 

Endurance 

Cycle  Time 

Operating 
Range  from 
Ship 

Technical  Maturity 
or  Risk 

Communication 

Robustness 

Interoperability 

Flexibility  & 
Adaptibility 

Logistics  & 
Readiness 
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SYSTEM  ENGINEERING 
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HANDLING 


Herb  Hecht 
'SoHaR  Incorporated 
cJyer  Cityr  California 


They  were  riotTested  fider;the  exception 
conditions 

T4e|pe£|uirennents  werenot  specific  aboiit 
exceptions  that  had  to  be  tolerated 

Comprehensive  specification  of  exceptions 
that  have  to  be  tolerated  is  difficult  -  or  is  it 
impossible11? 


owsoPwIpE  fIls 


f‘T£fe  main  Ij n e  softw^  code^usu al ry  does 


m  nr 


r.r  w 


VC) 


u  r  wnon 


the  software  exception  code  does  not 


properly  handle  abnormal  input  or 


i 

-  or  when  an 

interface  does  notrespond  inlhe 
anticipated  or-desired  manner.” 

C.  K.  JJansets/The  Status  of  Reliability  Engineering  Technology  2001, 
Newsletter  of  the  lEfeRReliability  Society,  January  200  r  / 


SOME  SPECTACULARS 

mt«R  RAC-25  li?AL  radiation  ov|rdoses 

.  DID  NOT  SUPPRESS  OPERATOR  INPUT  WHILE 

maqneIb  yviRE  r|posj|ionejd-  " 

•JKrPSsiRs  crashed  aftFr  launch 

^  d|sabl|d  Bang u age  providedIxc'handl. 

.  permPFed  sWr-DOWN  O0BOTH  nav  SYST. 

■IS5Ir§.  po|P\R  lander  kard  landed 
JaLurejo  de-bounce  contacts 


iPomHlcE  of  exceptor 

HANDLING  Wr 


0.15 


□  RECOVERY 

■  PROCESSING 
ERRORS 

□  HARDWARE 

□  SOFTWARE 


Toy,  W.  N.,  “Fault-Tolerant  Design  of  AT&T  Telephone  Switching 
Systems”  in  Reliable  Computer  Systems:  design  and  evaluation, 
Siewiorek  and  Swarz,  eds.,  Digital  Press,  Burlington  MA,  1992 


MPORTANCE  OF  EXCEPTION 

HANDLING  -  2 

ALL  FAILURES  GLOBAL  FAILURES 


Kanoun,  K.  and  T.  Sabourin,  “Software  Dependability  of  a  Telephone 
Switching  System”,  Digest  of  Papers,  FTCS-17,  Pittsburgh  PA,  July  1987, 
pp.  236  -  241 


EXCEPTION  HANDLING  AND 

HHRRnMin' 


Fraction  EH 


Hecht,  H.  and  P.  Crane,  “Rare  Conditions  and  their  Effect  on  Software 
Failures”,  Proc.  of  the  1994  Annual  Reliability  and  Maintainability 
Symposium”,  January  1994,  pp.  334  -  337. 


QUOTES 


*ftie  main  line  software  code  usually  does  its|ob.  Breakdowns  typically 
oceyf  when  the  software  exceptiorf  code  doesmot  properly  handle 
^bnorrhSl^nput  or  environmental  conditions  -  or  when  an  interface 
does  not  respond  in  the  anticipated  or  desired  manner.” 


C.  K.  Hansen^Tfoe  Status  of  Reliability  Engineering  Technology  2001 ,  Newsletter  of  the  IEEE 
Reliability  Society,  January  2001 


Piperefore  tie  identification  and  handling  of  the  exceptional  situations 
that  might  occuf  is  often  just  as  (un)relinble  as  human  intuition.” 

Flaviu  Cristianv“Exception  Handling  and  Tolerance  of  Software  Faults”  in  Software  Fault  Tolerance, 
Michael  RflLyu,  ed.;  Wiley,  New  York,  1995 


FAILURES? 


WHY 

h  THlfpROGRAMS  WERE  NOT  TESTED 
."UNDER  TH'E-CONDl.T-ioNS  THAT 
CAUSED  TH^FAipjRES.  -  ' 

•  THBRE  WERE  NO  REQUIREMENTS 
■FOR  TESTING  UNDER  tKsE  ' 
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•GENERAJINO  REQUIREMENTS  FOR 
EXCEPTION  HANDLING  IS  DIFFICULT 


-  EXCEPTION  CONDITIONS  ARISE 
pRoS SEVERAL  LEV-ELS-  .  - ' 

•j  EXCEPTION  GONDITIONSTCRE  more' 
^DIFFICuIt  TO  UNDERSTAND  THAN 
R&GN  LINtREQUIREMENfs 

•'exqBtions  OCCUR  INFREQUENTLY 

^BUT  REQUIRE  DISPROPORTIONATE 
EFFORT 


SOURCES  OF  EXCEPTIONS 

OPERATIONAL  REQUIREMENTS 

LOSS  OF  POWER,  COMMUNICATION,  THERMAL  CONTROL 

IMPLEMENTATION  DETAIL 

CALIBRATION  ANOMALIES,  ACTUATOR  STATES,  OPERATOR  INPUT 

COMPUTING  ENVIRONMENT 

HARDWARE  FAILURES,  MEMORY  ERRORS,  EXECUTIVE,  MIDDLEWARE 

MONITORING  AND  SELF-TEST 

OVER-TEMPERATURE  SENSORS,  SYSTEM  PERFORMANCE  TEST 

APPLICATION  SOFTWARE 


ASSERTIONS,  VIOLATION  OF  TIMING  CONSTRAINTS,  MODE  CHANGES 
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OPERATIONAL  REQUIREMEN 
IMPLEMENTATION  DETAILS 
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Process  Improvement  Background:  Circa  1910 


SE I  Partner 


Transdyne  Corporation 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


PreCMMI  History  and  Influences 


*  The  history  of  process  improvement  has  origins  back  to  the  turn  of  the 
century  during  the  American  industrial  age* 

The  establishment  of  assembly  lines  by  Henry  Ford  caused  a  demand 
for  skilled  workers. 

*  The  early  assembly  lines  were  plagued  with  quality  problems  which 
were  not  discovered  usually  until  the  final  part  was  inserted  into  the 
Model  T  Ford. 

*  The  scrap  pile  was  substantial,  which  increased  the  cost  to  the 
consumer. 

*  An  additional  quality  problem  was  detected  in  the  manufacturing  of  gun 
casings  in  WWI  that  exploded  upon  firing  and  caused  casualties* 

*  Faced  with  these  and  other  manufacturing  quality  control  challenges, 
early  pioneers  of  process  improvement,  such  as  Joseph  Juran,  Walter 
Shewhart,  W,  Edwards  Deming,  Phil  Crosby  and  !ater  workers,  such  as 
Watts  Humphrey  of  the  Software  Engineering  institute  decided  to  focus  on  the 
process  and  not  just  inspecting  the  products. 


CMMI 
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CMMI  vl  .2:  Process  Improvement  Model  Heritage 


Transdyne  Corporation 
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CMMI  Implementations  in 

SEI  Partner  Small  &  Medium  Organizations 


Key  Process  Model 

Timeline 

S/W  CMM 

1995 

S/W  CMM  v2.0 

Never  released 

System  Engineering  (SE)  CMM 

1995 

Integrated  Product  Development  (IPD) 
CMM 

1997 

Electronic  Industries  Association 
(EIA)  731  (Systems  Engineering) 

1998 

CMMIvl.1 

March,  2002 

CMMIvl.2  August,  2006 

•  The  heritage  of  CMMI  vl  .2  comes  from  numerous  ISO,  IEEE,  EIA 
and  SEI  models. 


CMAVf 


•  The  CMMI  is  an  integrated  model  from  EIA  731 , 

S/W  Capability  Maturity  Model  (CMM)  v2.0  and 
Integrated  Product  Development  Capability  Maturity  Model. 
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-  Note  of  Special  Importance:  The  Usefulness  of  Models 

SEIPartner 


Transdyne  Corporation 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


"All  models  are  wrong, 
but  some  are  useful.” 

George  Box 

(Quality  and  Statistics  Engineer) 

_ _ _ y 


•  A  CMMI  model  is  not  a 
process. 


•  A  CMMI  model  describes  the 
characteristics  of  effective 
processes. 


CMMI 


©2006  by  Carnegie  Mellon 
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CMMI  vl  .2  Benefits  for  Small  -  Medium 
SE  Services  and  Development  Organizations 


Transdyne  Corporation 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


CMMI 


Smalt  -  Medium  Organizations  Benefits 


The  CMMI  model  is  a  structured  set  of 
good  management  practices  collected 
from  practitioners  across  private  industry 
and  government  organizations. 


Implementation  of  CMMI  model  assists 

organizations  by  providing  processes  to: 

1 .  Understand  the  current 
organizational  maturity  and 
process  capabilities 

2 .  I mprov e  current  capabilities  to 
achieve  business  performance 
goals,  such  as  performance  and 
quality 

3.  Plan  and  implement 
improvements 


==-  Process  Improvement  Paradigm: 


Transdyne  Corporation 
http://transdvnecorp.com 


SEI Partner  Balancing  Resources  and  SE  Services  Business  Case  ™ SedTum organizations 


Process  Performance  Goals 


Schedule  estimation 
Requirements  volatility 
Customer  satisfaction 
In-time  technical  expertise 


Budget 


Available  reso 


Iff 


Low  labor  and  overhead 
Littfe  formal  process 
improvement  background 


7-10  technical 
staff  members 
Level  of  effort  staffing 
Go-located  at  customer 
facilities 


ations 


SE  services  projects 
Cycle  time:  1  -6  months 
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SE  Services  Perspective: 

SEIPartner  Overview  of  CMMI  vl  .2  Process  Areas  (PAs) 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Process  Area 


1.  Function 


2.  Purpose 


3.  implementation 


CMMI 


Benefits 


Process  Area  is  a  duster  of  related 
practices  in  an  area  that,  when 
implemented  collectively,  satisfy  a  set  of 
goals  considered  important  for  making 
improvement  in  that  area.  The  PAs  are 
used  as  building  blocks  to  construct  a 
foundation  for  improving  process 
performance. 

These  practices  provide  organizations  a 
set  of  proven  management  tools  that 
are  non-pi  escriptive  (never  a  set  of 
implementation  practices). 


Each  SE  services  organization  should 
determine  how  to  implement  these 
practices  within  their  organizations 
always  from  a  pragmatic,  “what  makes 
sense”  perspective. 
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-=^r  Avoiding  Confusion  on  the  Two  Model  Representations 
SEIPartner 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


The  same  22  Process  Areas 
are  arranged  in  2  different 
ways. 


Continuous  and  Staged  Representations 


The  continuous  and  staged  representations  provide 
two  views  into  the  SAME  data  base  of  information, 
the  22  CMMI  vl.2  Process  Areas. 
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Comparison  of  Continuous  and  Staged  Representations 
SEIPartner 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 
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CO 


implementation 

flexibility 

Maturity  Levels 

Capability  Levels 

satisfy  the  business 
goals 

provides  layers  in 
process  improvement 

pre-defined  sets  of 
process  areas 

Use  SCAMPI  appraisal 
methods 

Process  Areas  in 
Process  Categories 

Process  Areas  in 
Maturity  Levels 

Obtain  a  benchmark 


E^ 
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Ef 

E^ 

ET 
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0T 
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et 

eT 
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et 

eT 

E^ 
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^ —  SE  Services  Paradigm:  Participation  in  the  SE 

Vee  Activities  for  Small  to  Medium 
SEIPartner  organizations 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Types  of 
Roles 


Small  -  Medium  SE 
Services  Organizations 


Support  tasks  in  parts 
of  the  Vee 


Less  Involvement 
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Context  for  SE  Services  Background 
SEIPartner  Descriptions  and  Project  Examples 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


The  samples  of  SE  services 
background  descriptions  and 
examples  of  project  documents 
are  more  applicable  to  small  and 
medium  organizations  than 
larger  corporations. 


The  small  to  medium  SE  services 
organizations  often  function  as 
team  members  that  supply 
expertise  in  specialized  technical 
domains  or  provide  on-site 
technical  support,  such  as  in 
analysis,  verification  and  validation 
tasks. 

The  SE  services  background 
descriptions  and  examples  of 
project  documents  are  for  the 
Process  Areas  in  Maturity  Level  3, 
excluding  IPPD. 
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"="  SE  Services  Background  Descriptions  and  Examples 
SEI  Partner  Process  Category:  Process  Management 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Process  Category  Process  Areas 

Process  —  _  K 

Management  ^  y 

Organizational  Process  Focus 

Organizational  Process  Definition  +IPPD 

Organizational  Training 

Organizational  Process  Performance 

Organizational  Innovation  and  Deploy  ment 

Project 

Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management  +  IPPD 

Risk  Management 

Quantitative  Project  Management 

Engineering 

Requirements  Management 

Requ  i  rements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Support 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analy  sis 

Causal  Analysis  and  Resolution 

Decision  Analysis  and  Resolution 
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Process  Category:  Process  Management 


SEI  Partner 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

SE  services  practitioners 
rarely  participate  in  setting 
up  formal  process 
improvement  organizations, 
documenting  processes  and 
defining  process 
performance  measurements. 

Organizational 
Process  Focus 
(OPF) 

Documentation  of  participation  in  formal 
appraisals  (with  the  exception  of  ISO  audits)  or 
EIA  731  are  uncommon  in  these  environments. 
Organizational  business  plans  may  provide 
documentation  of  process  performance  goals, 
such  as  customer  satisfaction  or  improvements 
in  schedule  estimation. 

SE  services  practitioners 
often  use  work  aids,  such  as 
templates,  as  a  guide  to 
scheduling  tasks  in  their 
projects. 

Organizational 

Process 

Definition 

(OPD) 

Templates  often  are  used  to  document 
processes  for  small  -  medium  SE  services 
organizations.  These  templates  are  often  used 
to  plan  and  collect  performance  measurements, 
such  as  delivery  schedules,  hours  expended, 
action  items  and  review  attendance. 

SE  service  practitioners 
value  organizations  that 
provide  training  to  keep 
technical  skills  current. 

Organizational 
Training  (OT) 

EMAILS  or  announcements  of  existing  “brown 
bag”  sessions  or  presentations  by  invited 
technology  advocates. 
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"="  SE  Services  Background  Descriptions  and  Examples 
SEI  Partner  Process  Category:  Project  Management 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Process  Category  Process  Areas 

Process 

Management 

Organizational  Process  Focus 

Organizational  Process  Definition  +  IPPD 

Organizational  Training 

Organizational  Process  Performance 

Organizational  Innovation  and  Deployment 

Project  k 

Management  B  J 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management  +  IPPD 

Risk  Management 

Quantitative  Project  Management 

Engineering 

Requirements  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Support 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Causal  Analysis  and  Resolution 

Decision  Analysis  and  Resolution 

SEI  Partner 


Process  Category:  Project  Management 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background  Process  Area  Examples  of  Project  Documents 


SE  services  managers 
usually  are  familiar  with 


Project  Planning 
(PP) 


the  activities  in  these  PAs, 


Project  planning  information  if  often 
found  in  the  management  sections  of 
proposals  and  usually  contains: 


with  the  exception  of 


formal  risk  management. 


1 .  Estimation  of  LOE  staffing  and 
project  schedules. 


SE  services  managers  and 


practitioners  need  to 
accurately  estimate 
schedules,  including 


2.  Risk  identification  either  to  the 
cost  or  schedule  baselines  as 
opposed  to  technical  risks. 


adequate  preparation  time 


for  technical  reports  and 
review  packages. 


3.  Planning  for  specialized  technical 
knowledge  or  staff  willing  to  relocate 


4.  Planning  for  management  of 
technical  reports 


Action  item  tracking  is  a 
key  project  management 
task  as  the  majority  of  the 
action  items  often  directly 
impact  the  customer 


Project  Monitoring  & 
Control  (PMC) 


Progress  reports  and  technical 
review  packages  often  document 
tracking  and  resolution  of  customer 
sensitive  technical  issues. 
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SE I  Partner 


Process  Category:  Project  Management  (continued) 


Transdyne  Corporation 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

The  engineers  interact  with  the 
technical  points  of  contact  of 
suppliers,  frequently  as  team 
members.  Managers  are  tasked 
with  supplier  cost  and  schedule 
management  and  obtain  technical 
performance  status  from 
engineers. 

Supplier  Agreement 

Management 

(SAM) 

Progress  reports  showing  status  of  technical 
performance  and  acceptance  reports 
documenting  delivery  hardware  or  software 

Identified  risks  to  SE  services 
organization  typically  are 
assessed  to  the  cost  and  schedule 
baselines.  Technical  risk 
assessment  is  appropriate  if 
organization  is  providing  System 
Engineering  and  Technical 

Analysis  oversight  for  the 
customer. 

Risk  Management 
(RSKM) 

Progress  report  or  technical  review  packages 
showing  risk  evaluations,  using  appropriate 
classification  categories. 

SE  organizations  usually  do  not 
have  extensive  documented 
processes  unless  provided  by 
team  member  or  customer. 
Standard  work  environment  may 
be  determined  by  the  contract. 

Integrated  Project 
Management  (IPM) 

Technical  review  packages  or  progress  reports 
using  a  customer  or  team  member  provided 
templates. 
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SE  Services  Background  Descriptions  and 

SEI  Partner  Examples 

Process  Category:  Engineering 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Process  Category  Process  Areas 

Process 

Management 

Organizational  Process  Focus 

Organizational  Process  Definition  +  IPPD 

Organizational  Training 

Organizational  Process  Performance 

Organizational  Innovation  and  Deployment 

Project 

Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management  +  IPPD 

Risk  Management 

Quantitative  Project  Management 

Engineering  ^  ^ 

Requirements  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Support 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Causal  Analysis  and  Resolution 

Decision  Analysis  and  Resolution 

‘  * 

SEI  Partner 


Process  Category:  Engineering 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

SE  services  managers 

Requirements 

Visit  reports  and  minutes  of  technical 

and  engineers  usually  are 
directly  involved  with 
customers  in  developing 
technical  performance 
requirements.  As  a  team 
member,  the  engineers  are 
involved  in  defining 
operational  concepts  and 
performing  analysis  to 
balance  technical 
performance,  cost  and 
schedule. 

Development  (RD) 

meetings  with  customer  technical 
interchanges 

SE  services  managers  and 

Requirements 

Technical  progress  reports 

engineers  often  serve  as 

Management 

containing  information  describing 

members  of  change 
control  boards  and 
provide  significant 
contributions  to  tracking 
inconsistencies  and 
defects  to  manage 
requirements  changes. 

(REQM) 

inconsistencies  or  detected  defects 
in  requirements. 

Minutes  of  configuration  control 
boards  document  recommendations 
and  formal  changes. 
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SE I  Partner 


Process  Category:  Engineering  (continued) 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

SE  services  projects  are 
usually  focused  on 
providing  technical 
analysis  of  system 
functions  in  their 
specialized  domains. 

While  providing  technical 
support  for  customers, 
their  analysis  is  limited  to 
these  specific  functions. 

Technical  Solution 
(TS) 

Visit  reports  and  minutes  of  technical 
meetings  with  customer  technical 
interchanges. 

Progress  reports  often  provide 
excellent  examples  of  engineers 
participation  in  providing  technical 
performance  analysis. 

SE  services  organizations 
are  often  tasked  with 
specific  product 
integration  activities,  such 
as  conducting  readiness 
reviews  or  providing  on¬ 
site  support  at  the 
integration  facility. 

Product  Integration 
(PI) 

Technical  progress  reports 
containing  information  describing 
integration  status  as  well  as 
generated  action  items. 
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SEI  Partner 


Process  Category:  Engineering  (continued) 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

SE  services  engineers 
perform  verification  of 
requirements  and  designs 
in  their  specialized 
domains.  While  providing 
verification  resources  for 
customers,  their  testing 
and  analysis  is  limited  to 
the  specific  system 
functions. 

Verification  (VER) 

Visit  reports  and  minutes  of  technical 
meetings  with  customer  technical 
interchanges. 

Progress  reports  often  provide 
excellent  examples  of  participation  in 
the  different  verification  tasks 
(requirements,  design  and  testing). 

SE  services  organizations 
are  often  tasked  to  provide 
on-site  engineers  to 
develop  validation  plans 
or  to  conduct  or  witness 
these  tests,  with  specific 
product 

Validation  (VAL) 

Examples  of  technical  reports 
documenting  the  results  of  validation 
tests. 

Technical  progress  reports 
containing  information  describing 
integration  status  as  well  as 
generated  action  items. 
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"="  SE  Services  Background  Descriptions  and  Examples 
SEIPartner  Process  Category:  Support 


Transdyne  Corporation 
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CMMI  Implementations  in 
Small  &  Medium  Organizations 


Process  Category 

Process  Areas 

Process 

Management 

Organizational  Process  Focus 

Organizational  Process  Definition  +  IPPD 
Organizational  Training 

Organizational  Process  Performance 
Organizational  Innovation  and  Deployment 

Project 

Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management  +  IPPD 

Risk  Management 

Quantitative  Project  Management 

Engineering 

Requirements  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Support 

«=> 

Configuration  Management 

Process  and  Product  Quality  Assurance 
Measurement  and  Analysis 

Causal  Analysis  and  Resolution 

Decision  Analysis  and  Resolution 
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SEI  Partner 


Process  Category:  Support 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

SE  services  organization 

Configuration 

Copies  of  configuration  status 

typically  interface  to  CM 
systems  in  larger  projects 
or  may  be  tasked  to 
function  as  the  CM 
manager.  The  engineers 
often  are  members  of 
configuration  control 
boards  with  authority  in 
specialized  technical 
domains. 

Management  (CM) 

reports  showing  technical  points  of 
contact  for  controlled  documents. 

Copies  of  configuration  control  board 
meetings  and  action  items. 

SE  services  organizations 

Process  and 

Reports  containing  documentation  of 

typically  do  not  perform 

Product  Quality 

non-compliance  or  technical  reports 

“formal”  quality  assurance 
activities  for  their  projects. 
There  may  be  participation 
in  the  QA  activities 
performed  on  larger 
projects  Participation  by 
engineers  in  informal  peer 
reviews  is  a  more  frequent 
implementation  of 
objective  evaluation. 

Assurance  (PPQA) 

documenting  problems  detected 
during  “peer  reviews”. 

SEI  Partner 


Process  Category:  Support  (continued) 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


SE  Services  Background 

Process  Area 

Examples  of  Project  Documents 

SE  services  project 

Measurement  and 

Technical  progress  reports 

managers  typically  report 
cost  and  schedule 
performance 
measurements  as  part  of 
progress  reports  and 
status  reviews.  Technical 
performance 
measurements  are 
reported  while  defining 
and  refining  operational 
concepts  and  performing 
analysis  to  balance 
technical  performance, 
cost  and  schedule  or 
performance  testing. 

Analysis  (MA) 

containing  project  status  information 
or  analysis  of  planned  functional 
performance  or  actual  performance 
measurements  collected  during 
testing. 

Selection  of  alternative 

Decision  Analysis 

Technical  reports  documenting 

hardware  or  architectures 

and  Resolution 

selection  criteria  and  evaluation  of 

is  documented  in 
technical  reports,  usually 
as  “trade  studies”. 

(DAR) 

alternatives. 
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SEI  Partner 


Summary:  Comparison  of  CMMI  Implementation 
Success  Factors  and  Organization  Size 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


CMMI 

Implementation 
Success  Factors 


flatter  organization 


staff  receptive  ness  to 
new  ideas 


awareness  of  existing 
processes 

simpler  process 
performance  models 

process  variance 
simpler  to  control 


less  diversity  in 
products  and  services 
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efficient  communication 
skills 

flexible  processes 

depth  of  understanding 
of  the  business  goals 

staff  involvement 
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Small  &  medium  organizations  are  not 
“miniatures”  of  large  corporations! 


Smaller  organizations  provide  a  conducive 
environment  to  implement  CMMI  practices 
due  to: 

1 .  simplicity  of  organizational  structure 

2.  efficient  communications 

3.  staff  receptiveness  of  new  ideas 

4.  depth  of  awareness  of  the  processes 

5.  easier  to  minimize  variance  in 
performing  key  processes 
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The  End 


SEIParfner 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


You  have just  seen 

background  descriptions 
and  examples  of  project 
documents  for  SE 
services  organizations 
from  the  “30,000  feet" 
level. 


Questions  or  Comments  ? 
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Disclaimer 


The  Opinions  Expressed  by  the  Speaker 

Are  His  Own 
and 

Do  Not  Necessarily  Reflect  Anyone  Else’s 

..although 
They  Might! 


How  to  Paint  a  Room 


The  Role  of  Specs  &  Standards  in 
the  Systems  Engineering... 

..Business! 


Robert  B.  “Scott”  Kuhnen 
ASC/AFRL  Eng  Stds  Office 
Wright-Patterson  AFB  OH 
24  October  2007 


Shall  We  Get  Started? 


Not  so  fast!!! 


“Proper  interior  paint  preparation  of  your 
walls  and  ceilings  before  painting  will  often 
encompass  more  work  than  the  actual 
painting.  Up  to  75%  of  the  work  can  be 
getting  a  surface  ready  for  painting.” 

Karl  Crowder 

http://www.house-painting-info.com/index.html 


Tools  for  Prepping  Walls 


•  Safety  glasses  or  goggles 

•  Respirator  or  face  mask 

•  Ear  protectors 

•  Rubber  gloves 

•  Pry  bar 

•  Paint  scraper 

•  Wallpaper  steamer  (rent  if  needed) 

•  Can  opener  or  widening  tool 

•  Fan 

•  Hand  sanding  block 

•  Orbital  sander 

•  Screwdriver 

•  Putty  knife 

•  Sponge 

•  Cap  or  scarf 

•  Old  clothes 


Materials  for  Prepping  Walls 


Spackle  (compound) 

Fine-grit  sandpaper 
-  (100  -  120-grit  silicon  carbide) 


Detergent  and  ammonia  or  tri-sodium 
phosphate  (TSP) 

Self-adhesive  drywall  tape 

Primer  or  adhesive  pad 

Sizing  (for  wallpapering) 


Tools  for  Painting 


Drop  cloths 
Ladders 
Buckets 
Paint  edger 
Brushes,  4",  3",  and  11/2" 

Angled  sash  brushes,  1  1/2"  and  2" 
Roller  pan  with  screen 
Roller  covers  with  appropriate  naps 
Roller  handle 
Roller  extender 
Paint  guide 


Materials  for  Painting 


Masking  tape,  2"  wide 
Newspaper 

Adhesive  pad  or  primer 
Paint  thinner  (with  oil-based  paints) 
Aluminum  foil 
Rags 


nu^lES 


o 


What  the  experts  say... 


•  Most  people  think  they  know  how  to  paint,  and  usually  the 
results  are  pretty  good.  But  for  painting  contractor  John 
Dee,  "pretty  good"  isn't  good  enough.  After  nearly  three 
decades  of  rolling,  brushing,  and  spraying  paint  he  knows 
the  subtle  tricks  for  applying  smooth,  even  coats  to  walls, 
ceilings,  and  woodwork,  and  for  creating  crisp  boundaries 
between  colors. 


/^According  to  Dee,  there's  no  magic  to  getting  professional-^ 

looking  results.  Practice  helps,  and  thorough  surface 
preparation  is  essential.  But  the  key,  he  says,  is  to  paint  in 
an  orderly,  systematic  way.  So  whether  he's  painting  a 
multi-paneled  door  or  a  flat  expanse  of  wall,  he  proceeds 
almost  scientifically  from  one  step  to  the  next,  with  no 
shortcuts.  "Your  approach  to  the  task,  the  order  in  which  you 
do  things,  can  speed  the  work  or  slow  you  down,"  Dee  says. 


ENGINEERING  DESIGN  PROCESS 


There  are  lots  of  experts. . . 


•  “At  Mario’s  Painting,  we  believe  that  the 
secret  to  achieving  flawless-looking, 
beautiful  surfaces  both  inside  and  outside 

your  home  lies  in  the  pre-painting _ 

preparation.  Where  some  companies  may 
try  to  cut  costs  by  cutting  back  on  quality 
preparation  time,  we  put  in  a  full  day’s 
work  before  the  first  coat  of  primer  even 
goes  on  your  walls.” 


Preparing  the  surface  is  the  most 
important  part  of  any  painting  project.  If 
the  paint  doesn’t  have  a  smooth,  clean 
surface  to  adhere  Jo,  Jhe  result  will  be  a 
poor-quality  job  that  doesn’t  last  very  !on< 
“You  should  spend  at  least  as  much  time 
on  surface  prep  as  you  will  be  painting,” 
advises  Horst. 


If  it's  worth  doing,  it's  worth  doing  right  the  first 


time. 


Few  of 


us  really  realize  this,  or  even  MRe  to  admit  it, 
since  it  leads  to  more  work.  It  is  a  step  that  is  all 
too  often  left  out,  and  the  final  job  reflects  its 
^omission.  It  is  too  easy  just  to  start  painting  and-^ 

not  go  through  the  necessary  prep  steps. 

Indeed,  for  a  while  the  paint  job  may  even  look 
pretty  good.  But  sooner  or  later  the  poor  quality 
will  show  up. 

_ 


Talking  about  painting  or...SE? 


FIG  UK  I!  3.  I  he  ferns  L n^i nee i  in^  Process 


/^Systems  Analyst 
&  Control 
(Balance) 


fnireinents  Analysis  ^ 
AiiJfcie  Missions  &  Environments 
IdeutfiSFuncriourtL  Requirements  - 
Define  Rej^fiPerfanm  uce  & 

Design  C  oustreint  Requirements  y 


ts  Loop  Y 

P unc  rional  Analysis,.1  Alio  cation 
Decompose  to  Lower-Level  Functions 
Allocate  Ferfornmuce  &  Other  Limiting 
Requirements  to  .All  Functional  Levels 
Define  Refine  Fuucfionnl  Interface-.  ilnrecnnl  Eiteraal} 
Define  Refine  Inte^rnte  Fdnctmoal  Architecture  , 


Design  Loop 


5El«ct  Preferred 
AHenalh^ 

TradE-Gff  Studies 
Effectiveness  Analyses 
RhL:  Mann^ement 
C  onfL^nrnfion  Management 
Interface  Management 
Data  Management 
FerToimauce-Based 
Pr-ogress  Measurement 
■■  5EMS 
■■  TPM 

■■  Technical  Reiiews 


Synthesh 

Transform  Architectures  {Functional  to  FbyslcaL) 
Define  Alternative  System  C  nucepts, 

Configuration  Items  &  System  Elements 

Define  Refine  Physical  Interface;  (Internal  External} 

Define  Alternative  Product  &  Process  Solution; 


PROCESS  OUTPUT 


■  Decision  Data  Babe 

■■  Deri;ion  Support  Data 
■■  System  Functional  &  Physical 
Architectures 

■■  Specifications  &  Baselines 

■  Balanced  System  Mlurams 


XJVffl  966t-a  i  s- 1  i  w 


•  Defense  Specifications 

•  Defense  Standards 


•  Qualified  Products  Lists 

•  Non-Gov’t  Standards 

•  Int’l  Standards 


•  etc. 


£ 
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PROCESS  INPUT 

1  Customer  Needs.  Objective;.- 
Requirements 
"  Missions 

11  Measures  of  Effectiveness 
11  Environment; 

■■  Constraints 
1  Technology  Base 
1  Poor  Outputs 
1  Program  Decision 
Requirements 
■  Requirements  From  / 
Tailored  SperifLrarioias  / 

.uid  Standards  / 


Require  me  or?  Ad  aly-sb  N 
Analyse  Mission;  &  Environments 
Identify-  F  unc  tin  nal  Requirement':- 
Define  Refine  Performance  & 
Dengn  C  onstraint  Requirements 


U 


Systems  Analysis 
&  Control 
(Balance) 


Requirements  Loop 

F  unc  rional  AnaLysis/ADooation 
Decompose  to  Lower-Level  Functions 
.Allocate  Perforina  uce  A  Other  Limiting 
Requirements  tn  .All  Functional  Levels 
Define  Refine  Fnuctionnl  Interface;  (Internal-EitEnial" 
Define  Refine  Integrate  Functional  Architecture 

Design  Loop 


select  Preferred 
Alternatives 
Trade-Off  Studies 
Effectiveness  Analyses 
Risk  Management 
C  oufiguration  Management 
Interface  Management 
Data  Management 
Ferfonnauce-Based 
Progress  Measurement 
«  SEMS 
■■  TPM 

ei  Technical  Reliefs 


Verification 


SjTLfteSLS 

Transform  Architectures  [Functional  to  Physical) 
Define  .Alternative  System  Concepts, 

Configuration  Items  &  System  Elements 
Define  Refine  Fliyyca]  Interface;  (InternaTEitenial} 
Define  .Alternative  Product  &  Proces;  Solution; 


PROCESS  OUTPUT 


■  Decision  Data  Ba;e 

+|  Dertiion  Support  Da  ta 
■■  System  Fnnc  tic  ual  &  Physical 
.Architectures 

■■  Spec  ificatio  us  &  Baseline; 

■  Balanced  System  Solutions 


MIL-STD--IWB  UHAFT 
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Discussion 
Non- Attribution 


What  are  we  missing? 
Is  SE  Important? 


Are  we  on  the  right  track? 


.f«  <*>'n9 

A' 


Top  Five  Systems  Engineering 


I  ssues 


•  Lack  of  awareness  of  the  importance,  value,  timing, 
accountability,  and  organizational  structure  of  SE  on 
programs 

•  Adequate,  qualified  resources  are  generally  not  available 
within  government  and  industry  for  allocation  on  major 
programs 

•  Insufficient  SE  tools  and  environments  to  effectively  execute 
SE  on  programs 

•  Poor  initial  program  formulation 

•  Requirements  definition,  development,  and  management  is 
not  applied  consistently  and  effectively 

NDIA  Study  in  January  2003 


DoD  Systems  Engineering 

Shortfalls* 


•  Root  cause  of  failures  on  programs  include: 

-  Inadequate  understanding  of  requirements 

-  Lack  of  systems  engineering  discipline,  authority,  and 
resources 

-Lack  of  technical  planning  and  oversight 
-Stovepipe  developments  with  late  integration 

-  Lack  of  subject  matter  expertise 
-Availability  of  systems  integration  facilities 

-  Low  visibility  of  software  risk 
-Technology  maturity  overestimated 

Major  contributors  to  poor  program  performance 


*  DoD-directed  Studies/Reviews 


Are  We  on  the  Right  Track? 


•  Study  Findings 


•  Programs/SEPs 


-  Inadequate  understanding  of - ► 

requirements 


-  Lack  of  SE  discipline, 
authority,  and  resources 


-  Lack  of  technical  planning  - ► 

and  oversight 

-  Stovepipe  developments  with 
late  integration 


-  Lack  of  subject  matter 
expertise  at  integration  level 


Incomplete  discussion  of 
program  requirements 

Minimal  discussion  of 
technical  authority  and  IPTs 

Incomplete  technical  baseline 
approach 

Incomplete  discussion  of 
technical  reviews 

Integration  of  SEP  sections 


Strong  correlation  between  initial  findings  and 
SEP  and  Program  Support  findings 


Could  the  problem  be. 


Vol.  5.  No.  9  rtw  lagtftMrmg  Newtpopor  for  Worldwid*  Mil/Aoro  Electronics  Ifxluitry  Aufuit  1994 


Perry  scraps  mil-specs 

Rv  ttr.ux-  Raynrr 

WfViHTNOTON,  DC  -In  late  )unc. 

Detent  Secretary  William  IVrry 
ordered  a  dramatic  ah»mt  face  in 
the  I  Xjtense  Pcpartincm**  use  ni 
military  specification*  ami  *tan 
dard*  ordering  that  all  1>jD  pm 
gram*  rely  more  heavily  on  com 
mortal  pans  and  athipt  a  perfor¬ 
mance  Kilt'd  spccificatKtn  pmccfft 

While  Perry  s  announcement 
was  widely  anticipated  and  pub¬ 
licly  .ippUudcd  by  the  defense  elec 
tronic*  industry,  many  company 
officials  arc  concerned  that  the 
change  will  increase  uncertainty 
in  the  acquisition  process  and 
threaten  some  existing  systems 
that  are  operating  well  such  as  tile 
(Qualified  Manufacturing  Line 
iOMIl  a  DutVspceiftc  system  for 
certifying  a  rTunuiacniring  process. 

‘’Right  now  it  is  a  wait-and-see 
game."  says  Brad  Paulsen,  director 
of  marketing  for  military  and 
aerospace  products  at  National 
Sematt/nductur  ,  Sanu  Clara,  CAi 
"There  arc  a  lot  of  issues  that 


have  not  been  clarified" 

The  directive,  which  will  be 
phased  in  over  the  next  six 
months,  mandates  that  all  l>oO 
procurement  con 
tram  use  commercial 
and  industrial  specs 
and  standards  where 
they  exist,  the  u*c  of 
mil  aspect  witt  require 
a  waiver  Radiation 

Secretary  of  Defense 
William  Perry  has  introdaetd 
far  reaching  changes  to  the 
procurement  process, 
including  mandating  the 
use  of  performance  specs. 

hardened  components 
art*  exempt  from  the 
directive. 

In  another  major 
change,  program  man 
offers  nr  now  require! 
to  adopt  prrfnmtanu* 
based  spccitications 
fr>r  new*  systems  and 


major  m<  ximt.uions  The  pertor 
maucc  specs  describe  how  a  svn 
tem  is  to  runcitrm  bur  leaves  the 

/CmtmueJ  <m  page  32/ 


DoD  has  adopted 


Systems  engineering  is  an  interdisciplinary  approach 
encompassing  the  entire  technical  effort  to  evolve  and  verify  an 
integrated  and  total  life-cycle  balanced  set  of  system,  people, 
and  process  solutions  that  satisfy  customer  needs.  Systems 
engineering  is  the  integrating  mechanism  across  the  technical 
efforts  related  to  the  development,  manufacturing,  verification, 
deployment,  operations,  support,  disposal  of,  and  user  training  for 
systems  and  their  life  cycle  processes.  System  engineering 
develops  technical  information  to  support  the  program 
management  decision-making  process.  For  example,  systems 
engineers  manage  and  control  the  definition  and  management  of  the 
system  configuration  and  the  translation  of  the  system  definition  into 
work  breakdown  structures. 

Adopted  from  ANSI/EIA-632,  “Processes  for  Engineering  a  System" 


Systems  Engineering 
Fundamentals  from  Past  Programs 

SE  was  conducted  by  the  design  team 

-  Systemic  to  the  design  process 

-  Product  of  many  designs  by  the  same  teammates 
over  many  programs  and  many  years 

Common  Characteristics:  yesterday  and  today 

-  Small,  efficient  systems  engineering  staff 

•  Previous  design  engineers 
^7  ~  Knack  for  requirements  7^ 

^AppPGCIclUid  me  larger  challenge  at  the  system  level 

-  Not  always  collocated  and  not  always  the  same 
company 


Source:  Mr.  John  Griffin, 
former  ASC/EN  Director 


Unintended  consequences? 
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Perry  scraps  mil-specs 

Rv  ttr.ux-  Rayner 

W^uiTNCTON,  DC  -In  late  )urar. 

Defense  Secretary  William  IVrrv 
ordered  a  dramatic  ahmt  face  in 
the  I  )etensc  Department"*  uae  ci 
military  cpecification*  ami  *tan 
dif(  Jk  tmiering  that  all  DuD  pm 
gram*  rely  more  heavtjy  on  com 
menial  pans  arui  adopt  a  perfor¬ 
mance  based  specification  pnxvss 

While  Perry's  announcement 
was  widely  anticipated  and  pub¬ 
licly  appUudrd  by  the  defense  elec 
tromc*  industry,  many  company 
officials  arc  concerned  that  the 
change  will  ihcicasc  uncertainty 
in  the  acquisition  process  and 
threaten  some  existing  systems 
that  are  operating  well  such  as  the 
(Qualified  Manufacturing  Line 
i<)MU  a  l>)OspLTifit  system  for 
certifying  a  manufacturing  process. 

‘"'Right  now  it  is  a  wait-and-see 
game/'  says  Brad  Paulsen,  director 
of  marketing  for  military  and 
aerospace  product*  at  National 
Semiconductor  .Santa  Clara,  CAi 
"There  are  a  lot  of  issues  that 


have  not  been  clarified" 

The  directive,  which  will  be 
phased  in  over  the  next  six 
month*,  mandates  that  all  l>oO 
procurement  eon 
tracts  use  commercial 
and  industrial  specs 
and  standards  where 
they  exist,  the  u* c  of 
mil -specs  will  require 
a  waiver  Radiation 

Secretary  ol  Defense 
William  Perry  has  intredecert 
far  reaching  changes  to  the 
procurement  process, 
including  mandating  the 
use  of  performance  specs 

hardened  components 
are  exempt  from  the 
directive. 

In  another  major 
change,  program  man 
agers  ire  now  required 
to  adopt  prrfnmunu- 
based  specifications 
for  new  systems  and 


major  m<  ximtatuins  The  pertor 
mance  spec*  dc%cnK  how  a  svx 
tem  is  to  function  (hit  leaves  the 
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Unintended  Consequences  Abound 


Reexamining  Military 
Acquisition  Reform 

Are  We  There  Yet? 


Christopher  H.  Hanks,  Elliot  I.  Axel  band,  Shuna  Lindsay, 
Mohammed  Rehan  Malik,  Brett  D  Steele 


Prepared  tor  the  United  States  Army 

Approved  for  public  release;  distribution  unlimited 
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3  ARROYO  CENTER 


“While  the  report  is  Army¬ 
centric,  I  believe  the 
discussion  would  fit  all  of  the 
Services.  The  report  covers 
some  63  different  acquisition 
reform  initiatives,  some  of  the 
observations  related  to  Mil 
Specs...” 

“I  think  you  want  to  look  to 
where  we  need  to  be  headed 
in  the  future.” 


Steve  Lowell,  DSPO 


Examining  Acq  Reform... 

The  New  Acquisition  Environment  Could  Create  Ongoing  Problems 

For  many  of  the  interviewees,  some  of  the  acquisition  reforms  implemented  over  the  past 
decade  may  be  creating  an  environment  that  will  present  ongoing  problems.  A  deputy  PM 
(civilian)  said  that  the  switch  frnm  mil  cporc  tr>  a  pgrfnrmqnco-hasfid  approach  (in  which 
jmiLspees-crremoi  requirea  as  long  as  performance  levels  or  speciticaiions~afeH«eljLhas 
“meant  that  the  process  has  gone  from  "too  tight  to  too  fluffy."  The  use  of  "performance 
specs"  in  lieu  of  mil  specs  was  already  seen  to  be  leading  to  problems  with  contractors, 
who  are  given  a  larger  role  in  the  process.  On  the  one  hand,  contractors  "now  have  far 
“rfiore-freedom  to  get  into  trouble,"  as  one  individual  put  it  in  a  group  interview^-Ofrttie 
other  hand/sume  culili  actors  do  not  know  how  to  proceed  with  tins  newfreedom,  and 
could  have  trouble  "implementing  the  discipline  to  handle  their  new  responsibilities."  Many 
contractors  don't  like  the  performance-based  approach  because  of  the  uncertainty  it 
entails.  However,  others  are  profiting  from  the  new  "vagueness"  built  into  contracts.  One 
deputy  PEO  (civilian)  described  a  recent  experience  with  a  contractor:  "The  contract 
wanted  to  have  everything  quick,  so  it  was  vague,  and  now  [we're]  spending  dearly  for 
that  vagueness.  The  contractor  is  .  .  .  using  the  vagueness  to  do  changes-so  the 
vagueness  is  working  to  the  contractor's  benefit,  not  the  government's." 


One  deputy  PM  (civilian)  noted  that  the  performance-based  approach  is  not  even 
increasing  PM  flexibility.  Some  interviewees  mentioned  that,  without  mil  specs,  many 
Technical  Data  Packages  (TDPs)  are  not  being  updated  and  are  now  several  years  out  of 
date.  Some  interviewees  also  questioned  whether  the  reforms  were  really  saving  time  or 
only  shortening  some-pcocesses-wbi!e  lengthening  others.  A  PM  staffer  (civilian)  notsck-^ 
"Lots  of  regs  are  gone,  but  it's  not  clear  things  are  taking  less  time  as  a  result  because 
other,  different  things  are  taking  time  to  decide  because  we  don't  have  the  regs  and 
specs  to  fall  back  on  automatically.  We've  gone  from  "too  much"  to  "too  little." 
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Please  recite  with  me... 


Humpty  Dumpty  sat  on  a  wall, 
Humpty  Dumpty  had  a  great  fall. 
All  the  King’s  Horses  and 
all  the  King’s  men 
Couldn’t  put  Humpty 
together  again! 


WHAT 


ii  i 
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WHO 


Tool  Set 


Tool  Set  Tailored  to  Each  Center’s  Principal  End  Items 


Institutionalization  requires  infrastructure  to  maintain  and  update  policy  and  toolset 

consistent  with  evolving  acquisition  reform  initiatives 


EN  Technical  Processes 


Advanced  Tech 
Transition 

Requirements  Definition 


Integrated  Risk  Management 
Modeling  and  Simulation 


Allocation 


Configuration  Management 

Verification  “ 
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Mod 

Planning 

Concept 

Exploration 

Program  Definition  & 
Risk  Reduction 

Engineering  and  Manufacturing 
Development 

Production,  Fielding/Deployment 
&  Operational  Support 

Disposal 

Integrity 

Programs 


Operational  Safety,  Suitability  &  Effectiveness  Assurance 


Systems  Engineering  Revitalization 

Framework 


Driving  Technical  Excellence  into  Programs! 


Good  Systems  Engineering... 


You’ll  know  it  when  you  see  it? 

or... 


You’ll  know  it  only  after  you’ve  verified 
that  the  product  meets  the  specs  & 
standards  which  define  the  product? 


Fred  Rail  said... 


The  best  Statement  of  Work  contains  only 
three  words: 


“Meet  the  Spec!” 


Concept  Exploration 


Purpose:  evaluate  alternative  solutions 
to  the  initial  concept;  select  preferred 
solution 


Characterized  bv 

•  Competili 


vp  n; 


tAKm  i#J 


es 


Market  research  is  key. 


Analysis  of  Alternatives  (AoA) 
Work  guided  by  the  ICD* 


MNS  until  CJCSI  3170.01  is  revised 


Concept  and  Technology 

Development  Phase 

Key  Activities,  continued 


Develop  draft  performance  specification 


Identify  potential  environm 
consequences 


System  level  performance 
spec.  Guide  spec  may  be 
used  to  help  draft  system 
performance  spec. 


Meet  exit  criteria  for  C&TD  Phase 


Propose  exit  criteria  for  next  phase 


nnsn-PM  Npw  Pnlinv  -  33 


System  Development  &  Demonstration 

Phase 


4 

Pi? 


Standards  provide  proven 
solutions  to  reduce  risk. 


To  develop 


Reduce  program  risk 


Ensure  operational 
Ensure  design  f 
Assure  affordability 


Demonstrate  system  integration, 
interoperability,  and  utility 


System  Demonstration 


Purpose:  Demonstrate  the  ability  of  the 
system  to  operate  in  a  useful  way 
consistent  with  the  validated  KPPs. 


Kev  Activities: 


Conduct  extensiv 
operatj 


Interoperability  defined  by 
standards  is  a  key 
performance  parameter. 


-  Prepare  RFP  for  Low  Rate  Initial  Production 

-  Prepare  for  Milestone  C 

-  Update:  Information  requirements 


Production  &  Deployment 

Phase 


LRIP/IOT&E 


Ente 


Full-Rate 
Production  & 
Deployment 


FRP 

Decision 

Review 

SI 


Full-Rate  Production  & 
Deployment 


Testing  &  evaluation  conducted  to 
ensure  conformance  to 
performance  specs  and 
interoperability  standards  for  full 
rate  production. 


suita 


production 


LRIP  (OSD  T&E 
ograms)  and  LFT&E 
d  systems) 
ongress 

Full  rate  production, 
stem.  Start  support. 
xit:  Full  operational  capability; 


deployment  compete 
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Operations  and  Support  Phase* 


•  Emphasis  shifts  from  design/development 
engineering  to  supporting  the  fielded  system 


life 


*  Overlaps  Production  and  Deployment  Phase  since  items  are  deployed  prior  to  the  end 
of  production,  and  must  be  sustained  in  the  field 
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Which  Standards? 


Def.  Stdzn  documents: 

Military 

NGS 

Total 

Preparing  Activity 

371 

363 

734 

(speaks  for  DoD) 

AF  Custodian 

6356 

2742 

9098 

(speaks  for  AF) 

AF  Review  Activity 

1140 

265 

1405 

(reviews  for  ASC) 

7867 

3370 

1 1 ,207 

Design  Handbooks  (17) 

-  Shipping  only  1-  and  2-series  documents  today  -  on  CD 

AF  Characteristics  Guides  (6) 

-  Shipping  only  -  have  only  begun  migration  to  CD 

Misc.  support  to  other  technical  docs  &  publications 

Bottom  Line:  Each  of  the  sectors  (Space,  Aeronautical 
Maritime... we  all  have  a  body  of  knowledge... standards. 


Joint  Service  Specification 

Guides 


JSSG-2000 
Air  System 


JSSG-2004 

Weapons 


JSSG-2001 
Air  Vehicle 


JSSG-2002 

Training 


JSSG-2003 
Support  Sys 


JSSG-2005 

Avionics 


JSSG-2006 

Structures 


JSSG-2007 

Engines 


JSSG-2008  JSSG-2009 

Vehicle  Control  Vehicle 
&  Mgmt  Subsystems 


JSSG-2010 

Crew 

Systems 


The  Bedrock  that  is  ASC 


Defense  Standardization  Program 

ASC/EN  is  responsible  for  development  and 
maintenance  of  Engineering  Standards  under 
Defense  Standardization  Program  (DSP) 

-  Mandated  by  Public  Law  82-436:  DoD  5000. 1&2;  DoDD 
4120.24;  DoD  4120.3-M;  AFPD  60-1;  AFI  60-101 

Wing  engineering  tailors  and  applies  standards 

-  Responsible  for  application  feedback  to 
ASC/EN,  who  cares  and  feeds  for  the  REO’s 

Industry  design  teams  also  use  MIL  specs  and 
standards 


It’s  part  of  your  day  job! 


April 


“Notional”  REO  Month 
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18 


25 


5 

INF  IMR 

12 


13 


7 

iPO  Support 

14 


15 


TDY -  Inti  Stdzn  Working  Group 


Int’fStdzn 

Pre-Brief 

16 

Trip  Report 


19 

20 

SAE  Syti 

21 

tposium,  S 

22 
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26 

27 

28 

29 

30 

Trip  Report 
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Systems  Engineering  “Engine 


* 
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PROCESS  INPUT 

■  Customer  NEedyObjectinfs,' 
Requirements 

■■  Missions 

«  Miea:ure:  of  Effectiveness 
11  Environment: 

11  C  on:  train  ts 
1  Technology  Base 

■  Prior  Outputs 
1  Program  Decision 

Requirements 
*  R_e*CL □  i r t* lii e cl t From 
Tailored  Specifications  j 


and  Standards 


1 


Requirements  Analysis  N 
Analyze  Mission:  &  Environments 
Identify  F  unc  ria  uai  Requirement':- 
Define  Refine  Performance  & 

Design  C  onstraint  Requirements 

Requirements  Loop 

F unc  rion-il  Ana  ]y  ris/Allocation 

Decompose  to  Lower-Level  Functions 
Allocate  Performance  &  Other  Limiting 
Requirements  to.AU  Functional  Levels 
Define  REfine  Fuuctionnl  Interface;  (Internal-EitEniaJ: 
Define  Refine  Integrate  Fuuc  fioual  Arc  hitec  tiu'e 

Design  Loop 


select  Preferred 
Alternatives 
Trade-Off  Studies 
Effectiveness  Analyses 
Hhk  Management 
C  oufiguration  Management 
Interface  Management 
Da  to  Management 
Ferfoimauce-Based 
Progresa  Measurement 
«•  SEMS 
»  TPM 

11  Technical  Reviews 


Verification 


Syurbesis  ^ 

Transform  Architectures  {Functional  to  Physical) 
Define  Alternative  System  £  nuceptsf 
Configuration  Items  &  System  Elements 
Define  "Refine  Physic  a]  Interface:  (Interual.-Eitenial) 
Define  Alternative  Product  &  Process  Solution: 


4 


Feedback  Loop 


PROCESS  OUTPUT 

1  Decision  Data  Ba:e 
+|  Decision  Support  Data 
*■  System  Fnnc  tio  ual  &  Physical 
Architectures 

•*  Specifications  &.  Easeline: 

*  Balanc  ed  System  Salman  us 


T 

is; 
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Benefits  of  the  DSP 


Standards  are  “foundational”  to  all  that  we  do 

-  Measuring  program  execution,  success  and/or  failure 

-  Moving  both  the  State-of-the-Art  and  Tried-and-the-True 

-  Reducing  risk  in  programs  and  the  SE  process 

-  Providing  “confidence”  to  those  who  actually  execute 
the  SE  process 

-  Documenting  &  Communicating  Lessons  Learned 

-  Mentoring  the  Next  Generation 

-  Communicating  technologies  and  strategies  across 
entire  sectors... forming  a  common  understanding 

-  ..Shall  I  continue...? 


My  Assertion... 

•  Specs  &  Standards  are  not  gone! 

-We  are  “down  to”  only  12,000  in  the  aero  sector 

•  Spec  &  Standards,  and  all  the  work  it  takes  to 
create  them,  coordinate  them,  update  them, 
understand  them,  use  them,  is  “foundational”  to  the 

execution  of  the  SE  process  (not  a  “crutch!”) 

•  Development  of,  use  of,  translation  of  technical 
requirements  is  the  heart  of  the  technical  portion  of 

the  SE  process . as  we  revitalize  SE,  consider  the 

role  that  specifications  and  standards  play  in  the 
overall  “business”  of  systems  engineering. 


Now  then... let’s  paint  this  sucker! 


Back  Up  Material 


OSS&E  “Toolset” 


EN  Technical  Processes 


Advanced  T  ech 
Transition 

Requirements  Definition 


Integrated  Risk  Management 
■  Modeling  and  Simulation 


Configuration  Management 
Verification 


SOO/RFP  Contract  PDR  CDR 

i  O  O  O  eirt  — ► 


Mod 

Concept 

Program  Definition  & 

Engineering  and  Manufecturing 

Production,  Fielding/Deployment 

Planning 

Exploration 

Risk  Reduction 

Development 

&  Operational  Support 

Disposal 


Iff s tttution s tiling  OSSSE 


WHAT 

WHO 
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Air  Force  Policy  Directive 


Air  Force  Instruction 


Institutionalization  requires  infrastructure  to  maintain  and  update  policy  and  toolset 
consistent  with  evolving  acquisition  reform  initiatives 


I 


Headquarters  U.S.  Air  Force 


Integrity  -  Service  -  Excellence 


Air  Force  “Pre-Acquisition”  SE: 


Technical  Planning  and  Investment  to  Inform 

the  Decision-Making  Process 


Mr.  Terry  Jaggers,  SES 
Chief  Engineer 

Office  of  the  Assistant  Secretary  of  the  Air  Force 

(Acquisitions) 

23  February  2007 


Aero  Sector’s  JSSG’s 


The  JSSGs  assist  in  the  development  of  effective 
program-specific  specifications.  Such  specifications, 
which  define  the  expectations  for  the  product  and  the 
confirmation  those  expectations  are  met  throughout 
development,  form  the  basis  to  further  refine  product 
requirements,  the  significant  accomplishments  that  must 
be  achieved  throughout  development,  the  activities  and 
schedule  by  which  those  accomplishments  will  be 
achieved,  and  the  definition  of  the  work  to  be  performed 
in  the  conduct  of  those  activities.  Linking  the  product 
expectations  to  the  work  to  be  accomplished  in 
development  provides  the  basis  for  contracts  which  are 
both  executable  and  enforceable. 


A  Convergence  of  Technologies 

for  Better  Software 

NOW! 

Dottie  Acton 
Lockheed  Martin  IS&GS 
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Topics 


Two  categories  of  software  errors 
How  technologies  can  help 
Who  needs  to  be  involved 
Some  experiences 
Questions 

Backup  charts  with  technology 
descriptions 
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Two  Categories  of  SW  Errors 


Solving  the  problem  wrong 

-  Incorrect  implementation  of 
requirements  (off  by  one  bug, 
logic  errors,  etc) 

-  Bad  coding  practices  that 
leave  security  holes 

-  Interface  mismatches 

-  Delivering  too  late  to  be 
useful 

-  Un-maintainable  code 

-  Poorly  performing  software 

-  Poor  user  interfaces 


Solving  the  wrong  problem 

-  Misinterpretation  of 
requirements 

-  Missing  requirements 

-  Unused  capabilities 

-  Obsolete  requirements 

-  Failing  to  recognize  the 
existence  of  ‘wicked’  problems 
(the  solution  changes  the 
nature  of  the  problem) 

-  Focusing  on  generic  or 
supporting  domains  rather  than 
on  the  core  domain 


Copyright  2007  Lockheed  Martin  Corporation.  All  rights  reserved. 


3 


developer  customer 
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What  are  the  Technologies? 
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The  Good  News 


Some  technologies  help  with  the  issue  of  solving 

the  problem  wrong. 


Interface 

Mismatches 

Late  Delivery 

User  Interface 
Problems  | 

Incorrect 

k 

k, 

Security  Holes 

\ 

Maintenance 

Performance 

Impjementation 

\  Problems 

Problems 

Standards 

Desian  bv 

Checkers 

Contract 

Patterns  and 
Refactoring 


\  A 

TT7 

SOA  and 

Reuse 

Robust 

Tools 
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The  Better  News 


Some  technologies  help  with  the  more  difficult 
issue  of  solving  the  wrong  problem. 


Misinterpretation 
of  Requirements 

Missing 

Requirements 

Obsolete 

Requirements 

Wicked 

Problems 

Unused 

Capabilities 

Lack  of  Focus 
On  Core  Domain 
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developer  customer 

< - ► 


The  Best  News 

•  The  technologies  are  synergistic 


Domain-Driven  Design 

Ubiquitous  Language 

Deep  Models 

Early  Customer  Involvement 

Clear  Product  Vision 

Product  Backlog  User  Stories 

Continuous  Planning 

Time-Boxed  Development 

Demonstrations 

Anti-Corruption  Layer 

Bounded  Context 

Core  Domain 

Test-Driven  Development 

Automated  Builds 

Continuous  Integration 

Refactoring  Pair  Programming  Retrospectives 

Automated  Testing 

Performance  Monitors 

Refactoring  Browsers 

Test  Frameworks 

Standards  Checkers  Design  by  Contract  SOA 

Reuse  Robust  Tools 

•  The  barriers  to  adoption  are  relatively  low 

-  Good  FOSS  tool  support  to  get  started 

*  COTS  tools  also  available  with  additional  capabilities 

-  Adequate  literature  and  experience  base  for  training 
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Who  is  Involved? 


Who  is  involved  in  making  a  change  to  agile  and 
domain-driven  design? 

-  Everyone! 

•  Customers,  Users,  Systems  Engineering,  Software  Development, 
Test,  Specialty  and  Support  Disciplines,  Management,  etc. 

-  Software  developers  usually  like  the  change  because  it  is  a  more 
natural  way  to  solve  problems 

-  Systems  engineers  and  managers  sometimes  have  a  harder 
time  adapting 

•  Managers  fear  the  loss  of  the  perception  of  being  in  control 

•  Systems  engineers  sometimes  struggle  with  the  need  to  keep  the 
big  picture  in  mind  while  going  deeper  into  selected  areas  for  near- 
term  development 

Benefits  generally  outweigh  the  negatives 

-  Earlier  feedback  on  requirements,  architecture  and  design 

-  Earlier  visibility  into  problem  areas 

-  Good  vehicle  for  transferring  domain  knowledge 
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Some  Experiences  (1) 


•  Program  applying  many  of  the  practices 

-  Program  Characteristics: 

•  Mission  critical  technical  application,  critical  algorithms 

•  Adopted  both  domain  modeling  and  agile  practices 

•  Part  of  a  larger  system  doing  traditional  development 

-  Results: 

•  High  quality  product,  with  good  quality  measurements 

•  Able  to  make  substantial  change  to  add  capability  and 
improve  performance  with  very  little  impact 

•  Happy  engineering  team  and  happy  customers 

•  Practices  spreading  to  other  teams 
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Some  Experiences  (2) 

•  Program  applying  just  a  few  of  the  agile 
practices 

-  Program  characteristics 

•  New  capability  being  added  to  large  existing  system 

•  Many  new  developers 

•  Experts  still  involved  in  previous  release  that  was  behind 
schedule 

•  Focused  on  standards,  daily  status  meetings,  continuous 
integration,  refactoring,  automated  builds 

-  Results 

•  New  capability  developed  on  time  and  budget 

•  High  quality  code,  based  on  early  test  results  and 
independent  quality  assessment 

•  Happy  team  and  happy  customers 
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Summary 


•  Domain-driven  design  and  agile 
development  together  offer  substantial 
opportunities  for  improving  how  we  do 
business 

-Tool  support  is  now  robust  enough  to  support 
iterative  development 

-Adequate  material  is  available  for  training 

-  Substantial  and  sustained  improvements  are 
becoming  evident 
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Questions 
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Backup  Material 


•  Descriptions  of  the  Key  Technologies 

-  Description 

-  Benefits 

-  References 

•  Additional  descriptive  material 

•  A  book  for  further  study 
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Agile  Development 


•  Description: 

-  An  approach  to  software  development  that  uses  short,  time-boxed 
iterations  to  support  early  delivery  of  the  customer’s  highest  value 
capability 

•  Each  iteration  is  potentially  shippable 

-  Agile  approaches  use  continuous  planning,  analysis  and  design  rather 
than  completing  those  activities  up-front,  before  development  begins 

•  Customer  is  involved  in  prioritizing  and  clarifying  requirements  throughout 
each  iteration 

-  Examples: 

•  Scrum,  XP,  Disciplined  Agility,  FDD,  Adaptive  Project  Management 

•  Benefits: 

-  Produces  early  value  for  customers 

-  Accommodates  changing  requirements 

-  Improves  quality  and  productivity 

•  References: 

-  http://en.wikipedia.org/wiki/Agile  software  development 

-  Agile  Software  Development:  The  Cooperative  Game  by  Alistair 
Cockburn 
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Test-Driven  Development 

•  Description: 

-  An  approach  to  development  that  uses  tests  to  drive  the 
production  of  SW 

-  Write  a  test,  write  the  code,  run  the  test,  refactor 

-  Examples: 

•  Test  frameworks  include  the  xUnit  family  of  FOSS  tools  (JUnit, 
cppUnit,  nllnit,  etc)  as  well  as  commercially  available  tools 

•  Benefits: 

-  Produces  high  quality  code  with  good  interfaces  and  few 
dependencies,  which  improves  the  maintainability  of  the  system 

-  Makes  future  changes  easier  since  all  code  has  a  suite  of  tests 

•  References: 

-  http://en.wikipedia.org/wiki/Test-driven  development 

-  Test-Driven  Development:  By  Example  by  Kent  Beck 
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Continuous  Integration 


Description: 

-  With  continuous  integration,  developers  check  in  their  code 
several  times  a  day,  as  soon  as  they  complete  each  small  chunk 
of  functionality 

-  When  the  code  is  checked  in,  a  series  of  automated  tests  are  run 
to  ensure  that  both  the  new  code  and  the  existing  code  base 
function  as  expected 

Benefits: 

-  Defects  are  discovered  soon  after  they  are  introduced,  so  they 
are  easier  to  find  and  fix 

-  Fewer  unpleasant  surprises  late  in  development 

References: 

-  http://en.wikipedia.org/wiki/Continuous  integration 

-  Continuous  Integration:  Improving  Software  Quality  and 

Reducing  Risk  bv  Duvall,  Matyas  and  Glover 
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Time-Boxed  Development 

Description: 

-  Each  short  iteration  is  scheduled  for  a  specified  duration  - 
usually  from  2-4  weeks 

-  If  the  scheduled  work  cannot  be  completed,  it  is  deferred  to  the 
next  iteration 

Benefits: 

-  Establishes  a  project  rhythm  that  improves  productivity 

-  Provide  early  and  frequent  status  based  on  working  code 

-  Forces  hard  choices  about  capability 

•  The  content  must  be  allowed  to  change  since  schedule  and  quality 
are  fixed 

References: 

-  http://en.wikipedia.org/wiki/Timebox 

-  Agile  Project  Management:  Creating  Innovative  Products  bv  Jim 
Highsmith 
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Automated  Tests 


Description: 

-  Test  automation  can  occur  at  any  level  of  testing 

•  Unit  test  automation  is  supported  through  test  frameworks  like  JUnit 

•  Both  FOSS  and  COTS  products  are  available  for  automating  aspects  of  higher  level 
testing 

-  Automated  tests  are  designed  to  be  run  frequently,  so  they  must  be  fast  and  free 
of  side  effects 

•  Usually  automated  unit  tests  are  run  as  an  integral  part  of  development  (see  TDD)  and 
each  time  the  code  is  checked  into  the  CM  system 

Benefits 

-  Improved  feedback  to  the  developer  and  increases  the  quality  of  changes  to  the 
code 

-  Automated  tests  are  especially  valuable  for  regression  testing 

-  Provide  a  safety  net  for  future  changes 

References: 

-  http://en.wikipedia.org/wiki/Automated  testing 

-  http://www.iunit.org/ 

-  Fit  for  Developing  Software:  Framework  for  Integrated  Tests  by  Rick  Mugridge 
and  Ward  Cunningham 
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Early  Customer  Involvement 


Description: 

-  Customers  create  and  prioritize  items  in  the  product  backlog 

-  Daily  interaction  between  customer  and  developers  both  clarifies 
requirements  details  and  allows  for  the  deeper  understanding  that  helps 
manage  the  complexity  associated  with  most  domains 

-  Examples: 

•  Business  person  working  with  developers  to  clarify  requirements  for  a  payroll 
system  or  invoice  system 

•  Hardware  engineer  working  with  developers  to  clarify  interactions  between  new 
hardware  and  controlling  software 

•  Systems  engineers  working  with  developers  to  develop  algorithms  for  scheduling 
access  to  specific  resources 

Benefits: 

-  Reduces  the  need  to  capture  requirements  detail  early  in  the  life  cycle 

References: 

-  http://en.wikipedia.org/wiki/Extreme  Programming  Practices#Whole  team 

-  Extreme  Programming  Explained:  Embrace  Change  (Second  Edition)  by 
Kent  Beck  and  Cynthia  Andres 
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Demonstrate  Each  Iteration 


Description: 

-  At  the  end  of  each  2-4  week  iteration,  the  features  developed 
during  that  iteration  are  demonstrated  to  the  customer 

•  Customer  feedback  drives  priorities  for  future  iterations 

Benefits: 

-  Promotes  better  understanding  of  additional  or  different  needs 

-  Working  code  demonstrates  real  progress 

References: 

-  http://www.scrumforteamsvstem.com/ProcessGuidance/Process/ 

SprintReview.html 

-  Agile  Software  Development  with  Scrum  by  Ken  Schwaber  and 
Mike  Beedle 
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User  Stories 


Description: 

-  Short  descriptions  of  features  that  can  be  implemented  in  2  days  to  2 
weeks 

-  Three  pieces  to  a  user  story 

•  Short  written  description 

•  Conversations  about  the  story  to  flesh  out  the  details 

•  Tests  that  convey  and  document  details  and  that  can  be  used  to  determine 
when  a  story  is  complete 

Benefits: 

-  Provide  a  good  basis  for  estimating  size  as  well  as  a  mechanism  for 
understanding  user  needs 

-  Force  a  shift  to  verbal  communication  for  feature  details,  which  is  much 
higher  band-width  and  supports  rapid  feedback  cycles 

-  The  tests  associated  with  the  stories  provide  executable  documentation 
of  user  requirements 

References: 

-  http://en.wikipedia.org/wiki/User  story 

-  User  Stories  Applied  for  Agile  Software  Development  by  Mike  Cohn 
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Product  Backlog 


Description: 

-  A  product  backlog  is  a  prioritized  list  of  features  desired  for  a  product 

•  Grows  and  changes  over  time  as  more  is  learned 

•  Scope  line  shows  how  much  can  be  accomplished  with  current  funding 

-  Prioritization  is  primarily  based  on  customer  needs,  but  must  also 
consider  technical  dependencies 

-  Commercial  and  FOSS  tools  are  available  to  help  manage  both  the 
product  backlog  and  iteration  backlogs 

Benefits: 

-  Prioritized  development  of  functionality  maximizes  customer  value 

-  Allows  users  to  add,  subtract  or  change  features  based  on  new  needs 

-  Scope  line  provides  on-going  visibility  into  features  that  can  be 
developed  with  current  funding 

References: 

-  http://www.mountainqoatsoftware.com/product  backlog 

-  Scaling  Software  Agility:  Best  Practices  for  Large  Enterprises  by  Dean 
Leffingwell 


Copyright  2007  Lockheed  Martin  Corporation.  All  rights  reserved. 


22 


Frequent  Deliveries 

Description: 

-  Short  iterations  allow  early  delivery  to  customer 

•  Each  iteration  should  be  potentially  shippable 

-  Often  multiple  iterations  are  grouped  for  delivery  to  customer 

•  Functionality  is  complete  with  each  iteration,  but  there  may  be  need 
for  a  ‘hardening’  iteration  before  shipment  to  address  publication  of 
user  documentation,  final  system  tests,  etc 

Benefits: 

-  Allows  customers  to  get  early  benefit  from  the  system 

-  Use  of  the  system  in  the  customer  environment  gives  improved 
opportunities  to  discover  ‘the  real  requirements’ 

References: 

-  httD://www.stsc.hill.af.mil/CrossTalk/2002/10/mccabe.html 

-  Lean  Software  Development.  An  Agile  Tool  Kit  bv  Mary  and  Tom 
Poppendieck 
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Clear  Product  Vision 


Description: 

-  Product  vision  is  established  early  to  guide  future  development 
efforts 

•  Essential  when  requirements  are  not  fully  detailed  initially 

-  One  technique  to  establish  the  vision  is  to  design  the  ‘box’  for 
the  product 

•  Differences  between  boxes  designed  by  different  teams  can 
illuminate  areas  of  disagreement  on  product  priorities 

Benefits: 

-  Helps  focus  customers  and  developers  on  the  essentials  of  the 
product 

-  Guides  lower  level  implementation  decisions 

References: 

-  http://www.innovationaames.com/game/PRODUCTBOX.aspx 

-  Agile  Project  Management:  Creating  Innovative  Products  by  Jim 
Highsmith 
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Automated  Builds 


Description: 

-  Goal  is  to  reduce  the  build  process  to  a  simple  us-of-a-button 
action 

•  Every  programmer  can  perform  a  build  whenever  it  is  needed 

-  Incremental  builds  can  help  make  this  a  reality 

•  With  today’s  tools  it  is  possible  for  even  the  largest  systems  to  build 
multiple  times  a  day 

-  Enables  effective  Test-Driven  Development 

Benefits: 

-  Reduces  time  programmers  spend  on  repetitive  build  tasks 

-  Improves  ability  to  run  tests  for  every  change,  which  improves 
quality  and  productivity 

References: 

-  http://www.electric- 

cloud.com/solutions/aaile  software  development.php 

-  Integrating  Agile  Development  in  the  Real  World  by  Peter  Schuh 
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Pair  Programming 

Description: 

-  Two  individuals  work  side-by-side,  sharing  a  single 
workstation,  to  design  or  code 

-  Pairing  with  testers  and  engineers  can  be  beneficial 
when  requirements  clarification  is  needed 

Benefits: 

-  Provides  a  real-time  peer  review 

-  Good  mechanism  for  knowledge  transfer 

-  Improves  code  quality 

References: 

-  http://en.wikioedia.org/wiki/Pair  programming 

-  Pair  programming  Illuminated  bv  Laurie  Williams  and 
Robert  Kessler 
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Retrospectives 


Description: 

-  At  the  end  of  each  iteration,  each  team  meets  to  celebrate  the 
completion  of  the  iteration  and  to  capture  lessons  learned  for  the  next 
iteration 

-  Three  topics  to  consider 

•  What  worked  well  and  should  be  continued 

•  What  should  the  team  stop  doing 

•  What  needs  to  be  done  differently 

Benefits: 

-  Identifies  areas  for  improvement  in  the  next  iteration 

-  Improved  morale  when  team  members  know  that  the  are  listened  to 

References: 

-  http://www.retrospectives.com/ 

-  Agile  Retrospectives,  Making  Good  Teams  Great  by  Esther  Derby  and 
Diana  Lawson 
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Domain-Driven  Design 

Description: 

-  A  way  of  accelerating  software  projects  that  have  to  deal  with  complex 
domains 

-  Fundamental  principles 

•  The  primary  focus  should  be  on  the  domain  and  domain  logic 

•  Complex  domains  should  be  based  on  a  model 

-  Agile  approaches  enable  domain-driven  design 

•  Work  with  customers  to  develop  models  that  reflect  domain  concepts 

•  Provide  rapid  feedback  to  clarify  complex  areas 

Benefits: 

-  Deeper  understanding  of  the  domain 

-  Better  communication  between  developers  and  domain  experts 

-  Ability  to  make  breakthroughs  at  a  faster  pace 

References: 

-  http://domaindrivendesiqn.org/ 

-  Domain-Driven  Design:  Tackling  Complexity  in  the  Heart  of  Software  by 

Eric  Evans 
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Ubiquitous  Language 

Description: 

-  A  common  language,  based  on  the  domain  model,  that  serves  as  a 
communication  vehicle  between  engineers,  developers  and  domain 
specialists 

•  Use  of  model-based  terms  in  all  project  communication  facilitates  deeper 
understanding  of  the  domain  by  everyone 

•  One  of  the  best  ways  to  refine  a  model  is  to  explore  with  speech,  trying  out 
loud  various  constructs  from  possible  model  variations 

Benefits: 

-  Improved  communication,  which  results  in  better  models  and  better 
software 

-  Helps  in  the  discovery  of  hidden  concepts 

•  Often  these  arise  in  areas  where  the  language  does  not  flow  smoothly 

References: 

-  http://domaindrivendesiqn.org/discussion/messaqeboardarchive/Ubiauit 

ousLanquaqe.html 
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Deep  Models 


•  Description: 

-  An  incisive  expression  of  the  primary  concerns  of  the  domain 
experts  and  their  most  relevant  knowledge 

•  A  deep  model  sloughs  off  superficial  aspects  of  the  domain  and 
naive  interpretations 

•  A  deep  model  distills  the  most  essential  aspects  of  a  domain  into 
simple  elements  that  can  be  combined  to  solve  the  important 
problems  of  the  application 

•  Benefits: 

-  Deep  models  enable  acceleration  of  discovery  and  innovation 
within  a  domain 

-  Keeps  entire  project  on  the  same  page 

•  References: 

-  Domain-Driven  Design:  Tackling  Complexity  in  the  Heart  of 

Software  by  Eric  Evans 
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Core  Domain 


Description: 

-  The  distinctive  part  of  the  model,  central  to  the  user’s  goals,  that 
differentiates  the  application  and  makes  it  valuable 

•  Efforts  to  refine  and  distill  models  should  be  focused  on  the  core 
domain 

Benefits: 

-  Identification  of  core,  supporting  and  generic  domains  can  help 
drive  the  company’s  strategy  for  what  they  develop,  outsource  or 
purchase 

-  Helps  identify  the  impact  of  changes 

References: 

-  Domain-Driven  Design:  Tackling  Complexity  in  the  Heart  of 

Software  bv  Eric  Evans 
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Bounded  Context 


Description: 

-  Defines  the  scope  of  each  domain  model 

•  Identifies  what  has  to  be  consistent  and  what  can  be  developed 
independently 

•  Defines  the  boundaries  for  continuous  integration 

-  Relationships  between  contexts  can  take  multiple  forms 

•  Shared  kernel,  customer/supplier  development  teams  or  conformist 

Benefits: 

-  Clearly  identifies  boundaries,  which  improves  ability  to  integrate 
across  teams 

-  Understanding  of  relationships  between  contexts  drives 
appropriate  program  behavior 

References: 

-  Domain-Driven  Design:  Tackling  Complexity  in  the  Heart  of 

Software  by  Eric  Evans 


Copyright  2007  Lockheed  Martin  Corporation.  All  rights  reserved. 


32 


Anti-Corruption  Layer 


Description: 

-  Allows  new  models  to  interface  with  legacy  systems,  without 
losing  the  clarity  needed  for  deep  modeling 

-  Creates  an  isolation  layer  so  that  the  new  model  can  avoid 
corruption  caused  by  needing  to  adapt  to  the  semantics  of  the 
old  system 

-  Can  be  implemented  by  a  combination  of  fagade  and  adapter 
patterns,  but  it  is  more  sophisticated  than  either  of  those 

Benefits: 

-  Keeps  one  side  of  a  bounded  interface  from  leaking  into  the 
other,  so  the  new  models  are  not  corrupted 

•  Provides  a  translation  between  parts  of  the  system  that  adhere  to 
different  models 

References: 

-  http://domaindrivendesian.org/practitioner  reports/penq  sam  20 

07  06.pdf 
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Design  by  Contract 

•  Description: 

-  An  approach  to  software  design  that  makes  pre-  and  post¬ 
conditions  explicit  for  each  public  method 

•  Uses  asserts  (or  equivalent)  to  enforce  the  contracts 

-  Defensive  programming  is  used  at  system  boundaries,  but  not 
for  interfaces  within  a  boundary 

•  Benefits: 

-  Interface  mismatches  are  detected  immediately,  rather  than 
indirectly  through  the  errors  that  result  from  the  mismatch 

•  References: 

-  http://en.wikipedia.org/wiki/Desian  by  Contract 

-  Object-Oriented  Software  Construction,  Second  Edition,  by 
Bertrand  Meyer 
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Patterns 


Description: 

-  A  pattern  is  a  repeatable  solution  to  a  common  problem  in  software 
design 

•  Captures  lessons  learned  about  use  of  the  solution  in  various  situations 

-  Catalogs  of  patterns  exist  at  various  levels,  including  architecture 
patterns  and  design  patterns 

Benefits: 

-  Leads  to  higher  quality  designs  that  are  easier  to  maintain  with  fewer 
dependencies 

-  Enhanced  communication  of  intent,  based  on  the  pattern  selected 

•  The  pattern  name  conveys  a  lot  of  information  in  a  few  words 

References: 

-  http://en.wikipedia.org/wiki/Desiqn  pattern  %28computer  science%29 

-  Design  Patterns:  Elements  of  Reusable  Object  Oriented  Software  by 

Gamma,  Helm,  Johnson  and  Vlissides 
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Refactoring 


Description: 

-  An  approach  that  systematically  changes  the  internal  structure  of 
code  without  changing  its  behavior 

-  An  integral  part  of  test-driven  development 

-  Often  done  prior  to  making  major  changes  to  code,  in  order  to 
make  it  easier  to  make  the  changes 

•  Each  small  change  is  tested  so  any  errors  are  detected  immediately 

Benefits: 

-  Cleaner  code,  fewer  dependencies,  fewer  defects 

-  Supported  by  refactoring  browsers,  so  it  is  a  relatively  safe  way 
to  make  code  changes 

References: 

-  http://en.wikipedia.org/wiki/Refactorina 

-  Refactoring:  Improving  the  Design  of  Existing  Code  by  Martin 
Fowler 
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SOA  and  Reuse 


Description: 

-  SOA  is  an  architectural  style  where  existing  or  new 
functionalities  are  grouped  into  atomic  services 

•  The  goal  of  SOA  is  to  allow  fairly  large  chunks  of  functionality  to  be 
strung  together  to  form  ad-hoc  applications  which  are  built  almost 
entirely  from  existing  software  services 

Benefits: 

-  Supports  large-grained  reuse  of  independently  developed 
services 

•  Enhanced  productivity  and  quality  through  reuse 

-  Enables  increased  focus  on  the  core  domain 

References: 

-  http://en.wikipedia.org/wiki/Service  oriented  architecture 

-  Service-Oriented  Architecture:  Concepts.  Technology  and 

Design  by  Thomas  Erl 
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Robust  Tools 


Description: 

-  There  are  many  tools  available  today  that  actually  help  in  the 
development  of  software 

•  Standards  checkers,  and  standards  checking  services,  Refactoring 
browsers,  Test  frameworks,  Performance  monitors,  Modeling  tools, 
especially  those  with  reverse  engineering  capabilities 

-  Need  to  make  sure  that  the  selected  tools  do  not  drive  extra  work 

Benefits: 

-  Improved  developer  productivity 

-  Improved  quality 

-  Enablers  for  iterative  development  -  manual  processes  not  adequate  to 
support  short  development  cycles 

References: 

-  http://www.opensourcetestinq.org/functional.php 

-  http://www.opensourcetestinq.org/unit  iava.php 

-  http://www.ddi.com/TechSearch/searchResults.ihtml;isessionid=MPJGH 

OHFWKVKCQSNDLOSKHOCJUNN2JVN?quervText=tools 
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Lockheed  Martin  Aeronautics  Company 
Approach  to  Solving  Development 

Program  Issues 


John  E.  Weaver 
Christopher  L.  Blake 
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LM  Aero  Approach  to  Systemic  Development  Issues 


•  Industry  Trend  of  Performance  on  Aircraft 
Development  Programs 

•  What  is  in  the  Future 

•  What  LM  Aero  is  Doing 

•  Conclusions 
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History  of  Development  Performance 


DoD  --  "Since  2004, 
total  costs  for  a 
common  set  of  64 
major  weapon 
systems  under 
development  have 
grown  in  real  terms  by 
4.9%  per  year  -- 
costing  $165  billion 
($BY07)  more  in  2007 
than  planned  for  in 
2004“ 

GAO 
2007 

AF  -- 1.5  development  cost  growth  ratio  --  ongoing  programs  5  yrs 
beyond  M/S-B  -  No  improvement  in  3  decades 

RAND  2005 
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What  is  in  the  Future 


■7 


•  New  Military  Aircraft  are  Going  to  be  More  Complex. 

•  New  Aircraft  Development  Spans  are  Monotonically 
Increasing. 

•  Our  Future  Workforce  will  be  Less  Experienced  and 
More  Inclined  to  Change  Employers. 
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Aircraft  Are  Becoming  More  Complex 


Length  of  A/C  Development  Programs 
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Typical  Aerospace  Company  Age  Profile 


Relative 
Number  of 
Employees 


Median  Age:  Late 
40 ’s 


Most  Technical  Professionals  Over 
50  have  Worked  on  3  or  More 
Aircraft  Development  Programs 


Retirement 

Eligibility 
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Root  Causes  for  the  Performance 


■7 


•  Poor  Quality  Requirements  and  Requirements 
Management  Resulting  in  Designs  that  do  not  Fulfill 
Customer  Expectations 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Active  Management  of  Allocations 

•  Poor  Prior  to  M/S  B  Resulting  in 

Unrealistic  Schedules  and  Unexecutable  Plans 

•  Level  of  Detail 

•  Historical  Bases  for  Spans 

•  Linkage  of  Higher  and  Lower  Level  Planning  to  Key 
Integration  Events 

•  Interactively  Versus  Prescriptively  Determined  Key 
Program  Event  Dates 
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Root  Causes  for  the  Performance  -  Continued 


■7 


•  Limited  ixperience  of  Program  Technical  Personnel  and 
Ineffective  ommand  Media 

•  New  Inexperienced  IPT  Leads  are  Place  in  Critical 
Decision  Making  Roles  without  Adequate  Help. 

•  General,  High  Level  Command  Media  is  not  Readily 
Useable  by  People  Working  on  Development  Programs 

•  Inability  to  Effectively  and  Objectively 

,  Quality  and  Integrity  in  a  Timely  Manner 

•  Need  for  and  Type  of  Corrective  Action  is  Identified  Too 
Late  to  Avoid  Serious  Consequences 

•  Incomplete,  Inconsistent  and  Inappropriate  Metrics 
Incentivize  the  Wrong  Actions 
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What  Lockheed  Martin  Aeronautics  is  Doing 


■7 


•  Developing  a  Systematic  Method  to  Define,  with  the  Customer,  Functional 
Baseline  Requirements  Much  Earlier  in  the  Acquisition  Lifecycle 

•  Modeling  the  Aircraft  Development  Process  in  Sufficient  Detail  to  Identify  the 
Work  Products,  the  Sequence  in  which  they  are  Produced  and  the  Work 
Product  Handoffs 

•  Collecting  the  Best  Practice  Information  for  Creating  Each  Work  Product  and 
Making  this  Information  Available  to  Those  People  Working  on  Development 
Programs. 

•  Instituting  a  Process  to  Independently  Assess  the  Adequacy  of  Each  Work 
Product  Before  it  is  Released  and  Defining  Valid  Metrics  to  Assess  Real 
Performance  in  Every  Area  of  the  Program 
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System  Design 


Approach  Applies  to  Pre-contract,  Post-award 
Planning,  and  Program  Execution 
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System  Design 
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Define  Work  Sequence  &  Output 
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Refine  Work  Sequence  &  Output 


•  Technical  data 
management  function 

•  Gatekeeper  role 

•  An  independent  source 
of  performance  metrics 

•  Data  dissemination 
controls 


•  High  level  system  design 
using  a  standard 
methodology 

•  Scope  of  work  to  be  planned 


•  Standard  Tech  Plan  &  WPS 
provides  starting  point 

•  Top  -  level  definition  applied  to  the 
specific  program 


*  Lower  -  level  details  expanded  for 
program  execution 

•  Program  Technical  Plan  &  Program 
Work  Products  Standard 
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Technology  Development  Phase 


Define  Tier  1,  2  Architecture 
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Work  Flow  Captures  Tasks,  Sequence,  and  Work  Produ 


Process 


ionics  Development 


Procured  Subsystem  Development 


Conduct 

r 

Procurement 

— ► 

J 

V 

Requirement 

Definition 


Preliminary  & 
Detail  Design 


Process 


Requirements 
Definition 


Work  Tasks 


Fi 

Arcnnecrare 


Work  Products 


^Procurement 


WP 


Conduct 


Procurement 


Lockheed  Martin  Aeronautics  Cor? 


Training  System  Development 
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Standard  Plan  Provides  Sound  Basis  for  Program - 7 

Starting  in  Proposal  Phase 


TOP  LEVEL 
Program-Customized 
Technical  Development 
Framework 


Program 
Executes  to 
Integrated  Master 
Schedule  & 
EVMS 


Basis  for  Proposal 
Schedule  &  Estimate 
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Proposal  Contract  Award  Program 
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Technical  Integrity  in  the  Release  Process 
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LM  Aero  Approach  to  Systemic  Development  Issues — 7 

Conclusions 


•  In  Order  to  Remedy  Many  of  the  Problems  with 
Development  Programs,  the  Necessary  Top  Level 
Design  and  Planning  Must  be  Done  Before  M/S  B. 

•  In  Order  to  Function  with  Tomorrow’s  Workforce  in 
Tomorrow’s  Development  Environment,  Our  Industry 
Should  Take  a  Lesson  from  the  Commercial  World  and 
Make  Our  Development  Business  More  Turn  Key. 

-  Standard  Planning  Templates 

-  Standard  Processes  That  Produce  Standard 
Products. 

-  Command  Media  That  Define  The  Best  Practice  for 
Generating  the  Work  Product 
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SPAWAR 


Systems  Engineering  Program 

Systems  Engineering  Analysis  to  Improve  Concept 
Development  of  Complex  Defense  Systems 


▼ 


Michael  Harper  Dr.  Jerrell  Stracener 

Project  Engineer,  12 W  SE  Systems  Engineering  Program  Director 

SPAWAR  Systems  Center  Charleston _ Southern  Methodist  University 


Improving  operational  effectiveness  through  C4ISR  common  integrated  solutions 


ATTIC  Command  06142007 


SPMAR 

Center 

Charleston 


Purpose 


Define  the  framework  for  an  investigation  to  improve 
concept  development  of  new- start  and  reengineering 
of  complex  defense  systems  and  systems  of  systems. 


Formulate  Systems  Engineering  approaches  through 
systemic  analyses  to  provide  feedback  into  future 
policy,  guidance,  education,  and  training  updates  for 
Concept  Development  environment,  methodology, 
tools  and  skills  to  increasing  effectiveness  of  SE  in 
Concept  Development. 
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Current  Environment 


SfXWAR 

T 

Systems  Center 
Charleston 

Increasing  System  Complexity  (SoS) 

-Network  centric  and  extension  of  system 
applications  are  driving  more  integration 

-Functional  and  physical  interfaces  expanding  in 
number  and  complexity 

-New  approaches  to  testing  balanced  with 
modeling  and  simulation  must  match  new  system 
of  systems  requirements 

Experienced  but  Aging  Work  Force 

Not  sufficient  Systems  Engineering  education, 
research  and  training  resources  to  meet  needs 
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Issue 

Single  System 

System  of  System 

Constituents 

All  known  and  visible 

Changing,  potentially 
unknown 

May  not  know  is  part  of  SoS 

Purpose 

Predetermined  by 
system  owner  and 
conveyed  to 
constituents 

Continuously  evolving, 
cooperatively  determined, 
may  or  may  not  be  known  by 
systems  participating  in  SoS 

Control 

Hierarchically 

structured 

No  control  in  SoS 

Requirements 

Defined  and  managed 
by  System  Owner 

Often  required  to  anticipate 
how  system  will  be  used 

Ownership 

Pieces  developed  are 
owned,  maintained, 
and  evolved  by  owner 

Independently  owned, 
developed,  maintained,  and 
evolved 

Boundaries 

Closed  with  clearly 
defined  boundaries 

In  general,  unbounded  and 
part  of  a  larger  SoS 

Visibility 

Smith,  et  al.  SEI,  2006 

All  aspects  seen, 
understood  and 
controlled 

Components  and  process 
aspects  beyond  control  and 
visibility  of  developers,  users, 
and  owners 

SPAWAR 

Systems  Cenier 
Cnarieston 


Ob j  ective 


Develop  a  framework  for  identification  of  overlap, 
gaps  and  needs  based  on  current  and  evolving  DoD 
program  acquisition  policies  and  regulations 

-  Identified  to  determine  improvement  candidates 

Specific  focus  directed  at  earlier  “real”  consideration 
of  critical  elements 

-  Reliability,  Availability,  Logistics  (sustainment), 
Security  and  Disaster  Tolerance 

Directed  at  Aerospace  /  Defense  /  Security  sectors 
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Research  Framework 


sm VAR 

Cenier 

Cnarieston 


Vision 

Goal 

Objectives 


•Motivation 

•Needs 
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Methodology 


srmnan 

Systems  Center 
Cnarieston 


Specific  tasks  necessary  to  evolve  the  framework 

-Industry  and  government  needs  capture  and  assessment 

-  Identification  and  analysis  of  capabilities 
-Analysis  for  gaps  and  overlaps  with  respect  to  needs 
-Explore  and  define  alternatives  for  needs  response 

-  Evaluate  and  refine  alternates  to  evolve  preferred 
concept 

-  Strawman  research  framework  development  plan 
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SPAWAR 


A&D  SE  Research  Process  Concept 


Certfej 
Charleston 


DOD  &  Military 
Services  &  DHS 


Research  Program 
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SPAWAR 

Systems  Cert/er 


Critical  Elements  of 
Defense  Engineering  of  Complex  Systems 


Critical  elements  identified  for  engineering  of 
complex  defense  systems 

-SoS  Critical  Element  Reliability  and  Availability 

-SoS  Disaster  Tolerance 

-SoS  Security 

-Culture  and  Infrastructure 
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Critical  Element 

_ SoS  Reliability  and  Availability 


Center 

Cn<*rie$ton 


Metrics 


-Mean  Time  between  SoS  maintenance  and  support 
-Expected  Failure  free  SoS  operation  time:  MTBF 
-SoS  Mission  Success  Probability 
-Probability  of  SoS  being  ready  for  use 

Features  that  determine  SoS 


-Likelihood  of  being  ready  for  use  /  mission  success 
probability 

-  Life  cycle  cost  in  terms  of  product  and  customer  support 

Consequences  not  considering  R&A  as  critical  SoS 
elements 
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Critical  Element  SoS  Disaster  Tolerance 


SfXWAR 

T 

Systems  Center 
Charleston 

SoS  Disaster 

-Catastrophic  Failure  in  System  A  can  Result  due  to 
Missing  Requirement  in  System  B 

-Disaster  Tolerant  Driven  Requirements  Definition  MUST 
OCCUR  at  SoS  Level 

Enormous  SoS  Complexity  Necessitates 

-High-level,  Manageable  SoS  Model  with  “What-if  ’ 
Analysis  Capability  (SysML?) 

-Simple,  Robust  Injection  of  Failure  Models  in 
Component  SoS  Systems 

-Capability  to  Ensure  DT  in  Presence  of  Failure  Model 
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Critical  Element 


SoS  Security 


SfXWAR 

T 

Systems  Center 
Charleston 

Ensuring  access  and  functional  security  difficult  for 
single  component 

Exponentially  more  difficult  for  SoS 

-  Internet-based  access  architecture 

-  Shortened  development  and  deployment  cycles 

-  Integration  complexity 

-  Costly  and  time-consuming 

Model-Based  Security  Testing 
SysML  models  for  security  testing 

-  Attack  models 

-  Verification  vs.  validation 

-  Integration  security? 
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SPAWN*  Critical  Element 

Culture  and  Infrastructure 

Center 

Charleston 

Characteristics  of  DoD  and  Corporate  Culture 

-  Identify  cultural  norms  of  enterprises  (DoD  and  contractors) 

-  Understanding  cultural  norms  wrt  senior  management  decision  making 

-  Influencing  the  growth  of  enterprise  cultures 

Leveraging  DoD/  Contractor  Infrastructures  and  IP 

-  Understanding  competing  DoD  &  contractor  needs 

-  Leveraging/reuse  of  prior  design  to  shorten  development  timelines  (set-based 
concurrent  engineering) 

Role  of  Systems  Engineering  in  Enterprise 

-  Identification  of  SE  role  with  respect  to  PM  and  senior  management 

Role  of  Initiatives 

-  CMMI  (Effects  on  Corporate  Culture) 

-  Lean,  Six  Sigma  and  Collective  System  Design 
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Implementation 


Advancing  SE  Technology 
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Complex  Systems  Development 


Systems 

Development 


Technology 


SE  Policy  & 
Directives 

SE  Guidebooks 
&  Handbooks 
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SE  Standards 
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j«MMTCcenter  for  SE  -  Overview  Concept 


¥ 


Center 

Cnarieston 


Needs 

Resources 


Affiliates 

•AIA 

•AIAA 

•IEEE 

•INCOSE 

•NDIA 

•RMS 

Partnership 

•SAE 

Members 

•US  DoD 
•Military  Services 
•NASA 

•A&D  Contractors 
•A&D  Suppliers 
•Universities 

SE  Fellows 

•Industry 

•Government 

Graduate 

Students 

SE  Training  &  Consulting 

Subject  Matter  Experts 

Preferred  Providers 

Vision  & 
Leadership 


Products 


SE  Clearing  House 

Certification 

Technology  Development 
&  Insertion 

Lessons  Learned 

Policies,  Directives, 
Procedures  &  Plans 

Methods  &  Tools 

Handbooks,  Guidebooks 
&  Standards 

Processes 

Training  & 
Consulting 

Education  & 
Research 

Best  Practices 

Conferences 
&  Workshops 

A&D  Customers 


Work  Force 

Systems 

Development 

Systems 

Production 

Systems 

O&S/Sustainment 

Systems 

Acquisition 

University  Members 

-► 

Education 

Research 
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Summary 


•Vision,  Goal  and  Plan  Formulated 
•Research  Initiatives  Evolving 
•  Key  Meetings  Planned 
•Team  being  Expanded  -  Task  Driven 


Challenge  is  to  focus  resources  on  Concept 
Exploration  &  Definition  -  Not  Detailed  Design 
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Thank  you! 


SMU  -  EMIS 

Systems  Engineering  Program 


Questions? 
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Application  of 

Risk  Management  Practices  to 
NNSA  Tritium  Readiness  Subprogram 


National  Defense  Industrial  Association 

10th  Annual 

Systems  Engineering  Conference 
October  22-25,  2007 


Sham  K.  Shete’ 
Srini  Venkatesh 
Systems  Engineering 
Savannah  River  Site 
Washington  Savannah  River  Co. 

Aiken,  South  Carolina 


National  Nuclear  Security  Administration 


A  separately  organized  agency  within  the  U.S.  Department  of 
Energy 

Established  by  Congress  in  2000 

Responsible  for  enhancing  national  security  through  the  military 
application  of  nuclear  science 

Maintains  and  enhances  the  safety,  security,  reliability  and 
performance  of  the  U.S.  nuclear  weapons  stockpile  without  nuclear 
testing 

Works  to  reduce  global  danger  from  weapons  of  mass  destruction 

Provides  the  U.S.  Navy  with  safe  and  effective  nuclear  propulsion 

Responds  to  nuclear  and  radiological  emergencies  in  the  United 
States  and  abroad 


NNSA  Tritium  Readiness  Subprogram 

One  of  NNSA’s  missions  is  to  provide  tritium  to  the  US  nuclear 
stockpile. 

Tritium  Readiness  Subprogram  is  to  establish  a  system  that  can 
ensure  that  the  inventory  is  maintained  by  producing  new  tritium  to 
replace  that  tritium  lost  to  radioactive  decay  and  consumption. 

The  Tritium  Production  System  of  this  subprogram  will  produce 
tritium  by  irradiating  the  NNSA-designed  Tritium  Producing  Burnable 
Absorber  Rods  (TPBARs)  in  reactors  operated  by  the  Tennessee 
Valley  Authority  (TV A),  an  independent  government  agency. 

These  TPBARs  will  be  manufactured  commercially. 

After  irradiation,  the  radioactive  TPBARs  will  be  removed  from  the 
reactors  and  transported  to  a  new  Tritium  Extraction  Facility  (TEF)  at 
the  Savannah  River  Site  (SRS). 

There  the  tritium  will  be  removed  from  the  rods  using  a  special 
vacuum-thermal  process. 


Scope  of  TR  Subprogram  Risk  Assessment 


•An  Assessment  of  NNSA  Tritium  Readiness  Subprogram  risks  was 
conducted  as  part  of  the  Risk  Management  Process  adopted  by  the 
NNSA. 

•The  goal  of  this  overall  assessment  was  to  identify  risks  to  the 
Subprogram  and  to  develop  handling  strategies  with  specific  action 
items  that  could  be  scheduled  and  tracked  to  completion  in  order  to 
minimize  program  failures. 

•The  issues  and  assumptions  developed  during  the  assessment 
planning  stage  were  considered  during  several  meetings  by  a  team 
comprised  of  individuals  representing 

■  Pacific  Northwest  National  Laboratory  (PNNL), 

■  WesDyne, 

■  Kansas  City  Plant  (KCP), 

■  NNSA, 

■  NAC, 

■  Tennessee  Valley  Authority  (TV A),  and 

■  Savannah  River  Site  (SRS)  in  identifying  risks 


RISK  ASSESSMENT  PROCESS 


Risk  Grading  Guidelines 


Likelihood  (L) 

Criteria 

Non-Credible 

Determined  (through  formal  probability  calculations)  to  have  a  probability  of  occurrence  of  < 

10'6  (or  other  non-credible  probability  defined  for  the  activity) 

Very  Unlikely 

•Estimated  recurrence  interval  >  20  years  (or  perceived  life  of  program);  or 

•Will  not  likely  occur  anytime  in  the  life  cycle  of  the  Tritium  Readiness  Subprogram;  or 

•Estimated  recurrence  frequency  <  1  (i.e.,  event  not  expected  to  recur);  or 

•0%  <  Likelihood  of  single  event  occurrence  <  1 5%. 

Unlikely 

•Will  not  likely  occur  in  the  life  cycle  of  the  Tritium  Readiness  Subprogram;  or 
•10  years  <  Estimated  recurrence  interval  <  20  years;  or 

•1  <  Estimated  recurrence  frequency  <  2  (i.e.,  event  expected  to  recur  but  not  more  than 
once);  or 

•15%  <  Likelihood  of  single  event  occurrence  <  45%. 

Likely 

•May  occur  sometime  during  the  life  cycle  of  the  Tritium  Readiness  Subprogram;  or 
•5  years  <  Estimated  recurrence  interval  <  10  years;  or 

•2  <  Estimated  recurrence  frequency  <  5  (i.e.,  event  expected  to  recur  from  2  to  4  times);  or 
•45%  <  Likelihood  of  single  event  occurrence  <  75%. 

Likely  Likely 

•Will  likely  occur  sometime  during  the  life  cycle  of  the  Tritium  Readiness  Subprogram;  or 
•Estimated  recurrence  interval  <  5  years;  or 

•Estimated  recurrence  frequency  >  5  (i.e.,  event  expected  to  recur  more  than  five  times);  or 
•75%  <  Likelihood  of  single  event  occurrence  <  1 00%. 

Risk  Grading  Guidelines 


Consequence 

(C) 

Criteria 

Negligible 

•Minimal  consequences;  unimportant. 

•Some  potential  transfer  of  money  (<  $500K),  but  budget  estimates  not  exceeded. 

Negligible  impact  on  program;  minimal  potential  for  schedule  change;  compensated  by  available  schedule  float. 

Marginal 

•Small  reduction  in  Tritium  Readiness  Subprogram  technical  performance. 

•Moderate  threat  to  Tritium  Readiness  Subprogram  mission,  environment,  or  people;  may  require  minor  facility 
redesign  or  repair,  minor  environmental  remediation,  or  first  aid/minor  medical  intervention. 

•Cost  estimates  marginally  exceed  planned  budget  (>  $500K,  but  <  $1 M). 

•Minor  slip  in  schedule  (anything  less  than  3  months)  with  some  potential  adjustment  to  milestones  required. 

Significant 

•Significant  degradation  in  Tritium  Readiness  Subprogram  technical  performance. 

•Significant  threat  to  Tritium  Readiness  Subprogram  mission,  environment,  or  people;  requires  some  facility 
redesign  or  repair,  significant  environmental  remediation,  or  causes  injury  requiring  medical  treatment. 

•Cost  estimates  significantly  exceed  planned  budget  (>  $1 M,  but  <  $5M). 

•Significant  slip  in  schedule  (3  months  to  less  than  12  months)  with  resulting  milestones  changes  that  may  affect 
Tritium  Readiness  Subprogram  mission. 

Critical 

•Technical  goals  of  Tritium  Readiness  Subprogram  cannot  be  achieved. 

•Serious  threat  to  Tritium  Readiness  Subprogram  mission,  environment,  or  people;  possibly  completing  only  portions 
of  the  mission  or  requiring  major  facility  redesign  or  rebuilding,  extensive  environmental  remediation,  or  intensive 
medical  care  for  life-threatening  injury. 

•Cost  estimates  seriously  exceed  planned  budget  (>  $5M,  but  <  $10M). 

•Excessive  schedule  slip  (12  months  to  <  18  months)  unacceptably  affecting  overall  mission  of  Tritium  Readiness 
Subprogram  objectives,  etc. 

Crisis 

•Tritium  Readiness  Subprogram  cannot  be  completed. 

•Cost  estimates  unacceptably  exceed  planned  budget  (>  $10M). 

•Catastrophic  threat  to  program  mission;  possibly  causing  loss  of  mission. 

•Schedule  slips  >  18  months. 

Likelihood  (L) 


Risk  Grading  Matrix 


Very 

Likely 


Likely 


Unlikely 

Very 
Unlikely 

Non-Credible 


Low 

Moderate 

High 

High 

High 

Low 

Moderate 

Moderate 

High 

High 

Low 

Low 

Moderate 

Moderate 

High 

Low 

Low 

Low 

Moderate 

High 

Low 

Negligible 

Marginal 

Significant 

Critical 

Crisis 

Consequence  (C) 


* 


Normally  limited  to  assessing  residual  risks  with  Crisis  consequences 


Risk  Handling  Strategies 


TR  Subprogram  Risk  Assessment  Steps 


•  Identified  total  94  risks  events. 

•  Dispositioned  41  events  as  ‘combined  with  others’,  ‘deleted’,  and 
‘resolved’ 

•  Performed  Initial  Assessment  of  50  out  of  53  active  risk  events 

•  Documented  Assessment  in  the  Risk  Database/Risk  Form 

•  Identified  Risk  Handling  Strategies  and  Action  Items 

•  Performed  “Post-handling”  Assessment  of  residual  risks 

•  Performed  a  cost  contingency  analysis  using  “Crystal  Ball”  software 

•  Performed  Risk  Ranking  using  mean  cost  contingency 

•  Tracked  Risk  Handling  Strategy  Action  Items 

•  Reported  Risk  Status  during  Quarterly  Program  Review  meetings 

•  Re-assessed  TR  Subprogram  Risks  annually 


Risk  Assessment  Form 

HD  Number: 

Revision: 

Evert  Title: 

Type:  KM; 

Category: 

Assess.  Element: 

Title: 

Responsible  Org: 

Statement  of  Event: 

Likelihood: 

Basis: 

Consequence  i 

Ben  eft: 

Basis: 

Most  Significant  Cost  Impact  |Skj: 

Level: 

Event  Trigger 

Handling  strategy: 

DfriOfiption: 

Handling  Strategy  Action  Items: 

H£  Implementation 
Cost  (JK): 

BftiiS: 

H&  Umplemerttatifirt 
Schedule  jMosJ. 

Basis: 

Other  Handling  Strategics: 


Statement  of  Residual  Risk: 


Residual 

Likelihood: 

Basis: 

Residual 

Consequence: 

Basis: 

Residual  Ri  sk 

Level: 

fVtftdtmle 

Residual  Coil 

Impact 

Best  Case 

Most  Likely 

Worst  Case 

Residual  Schedule 
Impact  |!lo$K 

Impasted  Scope  of Work: 


Evaluation  Cm  me  nts: 


Evert  Comments: 


Last  Date  Evaluated: 


Status: 


Coriact: 


Date  Identified: 


Most  Significant  Schedule  Impact  |Mosj: 


Residual  Impact  Basil: 


Risk  Handling  Strategies  &  Their  Impact 


Avoid 


T  ransfer  0 


Mitigate  31 


Accept  1 3 


Risk  Level 

Initial 

Residual 

High 

21 

7 

Moderate 

22 

16 

Low 

7 

20 

Risk  Ranking  &  Cost  Contingency 


Ranking  Risk  Title  Mean  Mean-Total 

ID  Contingency  Contingency 

$K  $K  % 

1  40  Equipment  Design  Change  6,181.11  22,284  27.74 

2  38  Impacts  of  Costing  Factors  Outside  22,284 

Program's  Control  3,329.46  14.94 

3  77  Yield  Impacts  Production  Success  2,259.09  22,284  10.14 

4  8  Loss  of  Vendor  A  as  a  Long-Term  Supplier  2,162.99  22,284  9.71 

5  33  Equipment  Consolidation  Process  Design  1,746.17  22,284  7.84 

6  4  Loss  of  Vendor  B  as  a  Long-Term  Supplier  1,523.00  22,284  6.83 

7  23  Loss  of  Testing  Capability  800.46  22,284  3.59 

8  48  Unable  to  Reduce  Uncertainties  to  Meet  520.48  22,284  2.34 

Program  Needs 

9  41  Equipment  Performance  impact  506.85  22,284  2.27 

10  92  Excessive  impurities  in  Materials  493.01  22,284  2.21 

Total  Cost  Contingency 

Percentiles  Contingency 
_ ($K) 

60%  22,470 

80%  32,511 


Cumulative  Residual  Risk-Based  Cost 

Contingency 


Total  Risk 


>> 

o 

c 

0) 


a- 

© 


300 
250 
200 
150 
100 
50 
0 

748.36  12,889.14  25,029.91  37,170.69  49,311.46 


Benefits  of  Risk  Management  Process 


Quarterly  review  and  update  of  the  Risk  Management 
Database 

Risk  status  and  handling  strategy  action  item  tracking 
mechanism 

Generation  of  risk  handling  strategy  cost  &  schedule 
Generation  of  a  risk-based  cost  contingency  estimate 
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Purpose  of  this  Presentation 


To  show  how  Systems  Thinking  and  the  Systems  Archetypes  can  help 
to  avoid  common  counter-productive  behaviors  in  software  acquisition 
and  development  programs 

Agenda 

•  Systems  Thinking 

•  Feedback  Loops  and  Causal  Loop  Diagrams 

•  Selected  Systems  Archetypes 

—  Fixes  that  Fail 
—  Shifting  the  Burden 
—  Limits  to  Growth 

•  Selected  Software  Acquisition  and  Development  Archetypes 

—  Sacrificing  Quality 
—  Firefighting 
—  The  Bow  Wave  Effect 

•  Seeing  the  Bigger  Picture  and  Breaking  the  Pattern 
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Why  is  Software-Intensive  Acquisition  Hard? 


Complex  interactions  between  PMO,  contractors,  sponsors,  and  users 
Limited  visibility  into  progress  and  status — hard  to  comprehend 
Significant  delays  exist  between  applying  changes  and  seeing  results 
Unpredictable  and  unmanageable  progress  and  results 
Uncontrolled  escalation  of  situations  despite  best  management  efforts 
Linear  partitioning  (“Divide  and  conquer”)  isn’t  working  well 
Exponential  growth  of  interactions  as  size  grows  linearly 
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Acquisition  Programs  are  Dynamic  Systems 


Complex  Interactions:  Interactions  between  acquisition  stakeholders  are 
non-linear 

Non-linear  Behavior:  Non-linear  behavior  defies  traditional  mathematical 
analysis  because  of  the  presence  of  feedback 

Non-deterministic:  Complex  systems  are  not  deterministic 

Sensitivity  to  Initial  Conditions:  Results  may  vary  greatly  due  to 
seemingly  insignificant  differences  in  the  starting  point(s) 

Organizational:  Key  issues  in  software  acquisition  are  management  and 
organizational — not  technical 

Partitioning:  Not  possible  with  complex  interactions  between  components 
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What  is  Systems  Thinking? 


Systems  Thinking  developed  from  work  done  by  Jay  W.  Forrester  at  MIT 
while  modelling  electrical  feedback  effects 

•  Also  exists  in  economic,  political,  business,  and  organizational  behaviors 

Uses  feedback  loops  to  analyze  common  system  structures  that  either 
spin  out  of  control,  or  regulate  themselves 

Helps  identify  a  system’s  underlying  structure,  and  what  actions  will 
produce  which  results  (and  when) 

Systems  Thinking  teaches  us  that: 

•  System  behavior  is  greater  than  the  sum  of  component  behaviors 

•  “Quick  fix”  solutions  usually  have  side-effects  that  make  things  worse 

•  Improvement  comes  only  from  changing  the  underlying  system  structure 
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Causal  Loop  Diagrams  (CLDs) 


Depict  qualitative  “ influencing ”  relationships  (increasing  or  decreasing) 
and  time  delays  between  key  variables  that  describe  the  system 

Show  relationship  direction  by  labelling  them  Same  (+)  or  Opposite  (-) 
to  indicate  how  one  variable  behaves  based  on  the  previous  variable 

Consist  primarily  of  two  types  of  feedback  loops: 

•  Reinforcing  -  Changes  to  variables  reinforce,  moving  in  one  direction 

•  Balancing  -  Changes  to  variables  alternate,  achieving  equilibrium 
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Time  Delays 


Much  instability  and  unpredictability  of  systems  is  due  to  time  delays 
Time  delays  obscure  the  connections  in  cause-and-effect  relationships 

•  Side-by-side  causes  and  effects  would  be  “smoking  gun”  evidence 

People  are  inherently  poor  at  controlling  systems  with  substantial  time 
delays  between  cause  and  effect 

Examples: 

•  Over-steering  a  large  ship  that  is  slow  to  respond,  so  it  weaves  back 
and  forth 

•  A  thermostat  controlling  a  low-BTU  air  conditioner  that’s  slow  to  cool, 
so  the  house  temperature  bounces  between  too  hot  and  too  cold 

•  Inability  to  determine  which  surface,  handshake,  sneeze,  or  cough 
resulted  in  an  infection 
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What  are  the  Systems  Archetypes? 


The  Systems  Archetypes  depict  the  underlying  structures  of  a  set  of 
dynamic  behaviors  that  occur  in  organizations  throughout  the  world 

•  Each  causal  loop  diagram  tells  a  familiar,  recurring  story 

•  Each  describes  the  system  structure  that  causes  the  dynamic 

Archetypes  are  used  to: 

•  Identify  failure  patterns  as  they  develop  ( recognition ) 

•  Single  out  root  causes  ( diagnosis ) 

•  Engage  in  “big  picture”  thinking  ( avoid  oversimplification) 

•  Promote  shared  understanding  of  problems  ( build  consensus) 

•  Find  interventions  to  break  out  of  ongoing  dynamics  ( recovery ) 

•  Avoid  future  counter-productive  behaviors  (prevention) 
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Systems  Archetypes 


Over  10  recurring  “systems  archetypes”  have  been  identified,  including: 

Fixes  that  Fail 

•  A  quick  fix  for  a  problem  has  immediate  positive  results,  but  its 
unforeseen  long-term  consequences  worsen  the  problem. 

Shifting  the  Burden 

•  An  expedient  solution  temporarily  solves  a  problem,  but  its  repeated  use 
makes  it  harder  to  use  the  fundamental  solution. 

Limits  to  Growth 

•  Initially  rapid  growth  slows  because  of  an  inherent  capacity  limit  in  the 
system  that  worsens  with  growth. 
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“Fixes  That  Fail”  -  Systems  Archetype 


A  quick  Fix  for  a  Problem  Symptom 
has  immediate  positive  results,  but 
also  has  long-term  Unintended 
Consequences  that,  after  a  delay, 
worsen  the  original  Problem  Symptom 
as  the  Fix  is  used  more  often. 


based  on  Fixes  That  Fail 


— NDIA  Systems  Engineering  Conference 

Software  Engineering  institute  Carnegie  Mellon  bin  Novak,  October  24, 2007  10 

©  2006  Carnegie  Mellon  University 


Shifting  the  Burden” 


Delay 


Symptomatic 

Solution 


B1 


Problem 

Symptom 


B2 


Fundamental 

Solution 
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Systems  Archetype 


A  Symptomatic  Solution  temporarily 
solves  a  Problem  Symptom,  which 
later  recurs.  Its  repeated  use  over  the 
longer  term  has  Side-Effects  that  make 
it  less  and  less  feasible  to  use  the 
\  more  effective  Fundamental  Solution — 

\  trapping  the  organization  into  using 

\  only  the  Symptomatic  Solution. 

s  \  Impatience  with  the  delay  makes  the 

l  organization  choose  the  Symptomatic 

Side-Effect  Solution  in  the  first  place. 


Based  on  “Shifting  the  Burden” 
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“Limits  to  Growth”  -  Systems  Archetype 


Constraint 


Initially  rapid  growth  slows  because  of 
an  inherent  capacity  limit  in  the 
system  that  worsens  with  growth.  As 
greater  Efforts  produce  better 
Performance,  there  is  a  greater 
Limiting  Action  due  to  a  Constraint  in 
the  environment,  slowing 
Performance. 


Based  on  “Shifting  the  Burden ’ 
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Acquisition  Archetypes 


There  are  many  recurring  patterns  of  behavior  in  software  acquisition  and 
development  that  have  been  modelled  using  Systems  Archetypes  and  CLDs: 


Sacrificing  Quality 
Firefighting 

The  “Bow  Wave”  Effect 
Underbidding  the  Contract 
Shooting  the  Messenger 
Robbing  Peter  to  Pay  Paul 
Longer  Begets  Bigger 


•  The  90%  Syndrome 

•  Requirements  Scope  Creep 

•  Feeding  the  Sacred  Cow 

•  Brooks’  Law 

•  PMO  vs.  Contractor  Hostility 

•  Staff  Burnout  and  Turnover 

•  The  Improvement  Paradox 


— NDIA  Systems  Engineering  Conference 

Software  Engineering  Institute  Carnegie  Mellon  bin  Novak,  October  24, 2007  13 

©  2006  Carnegie  Mellon  University 


“Sacrificing  Quality”  -  Acquisition  Archetype 


As  schedule 
pressure 
increases... 

...and 
the  cycle  Schedule 
repeats  Pressure 
and 

worsens. 

...which 
increases 
schedule 
pressure... 


...quality  suffers... 
Quality 

...which  reduces 
errors. 


...and 
errors 
increase... 

Errors 


A 


B 


Rework 


y 


Available 
Resources  o 

However,  rework 
consumes  resources... 


..requiring 

more 

rework... 


As  schedule  pressure 
increases,  processes  are 
shortcut,  quality  suffers,  and 
errors  increase — requiring 
more  re-work.  However,  re¬ 
work  consumes  resources, 
which  increases  schedule 
pressure,  and  the  cycle 
repeats  and  worsens. 


based  on  “Fixes  That  Fail ” 


— NDIA  Systems  Engineering  Conference 

Software  Engineering  institute  Carnegie  Mellon  bim  Novak,  October  24, 2007  14 

©  2006  Carnegie  Mellon  University 


“Firefighting”  -  Acquisition  Archetype 


Early 

Development 
Activities  on 
Next  Release 


Resources 
Dedicated  to 
Next  Release 


Design 
Problems  in 
Current 
Release 

B 

Resources 
Dedicated  to 
Current 
Release 


Tolerance 

for 

Design 

Problems 


Problem 

Gap 


s 


If  a  design  problems  in  the  current 
release  are  higher  than  the  tolerance 
for  them,  more  resources  must  be 
dedicated  to  fix  them.  This  reduces 
problems,  but  now  fewer  resources 
can  work  on  the  next  release.  This 
undermines  its  early  development 
activities  which,  after  a  delay, 
increases  the  number  of  design 
problems  in  the  next  release. 


from  “ Past  the  Tipping  Point” 
based  on  ‘‘Fixes  That  Fail” 
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“Bow  Wave  Effect”  -  Acquisition  Archetype 


Risky  tasks  planned  for  an  early  spiral 
to  reduce  risk  are  postponed  to  a  later 
spiral,  making  near-term  performance 
look  better.  This  increases  risk  in 
subsequent  spirals  by  delaying 
required  risky  development  for  which 
there  is  now  less  available  schedule  to 
address  potential  issues,  and  less 
flexibility  in  the  system  to 
accommodate  changes  needed  to 
integrate  the  new  capability. 


based  on  “ Shifting  the  Burden 
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The  Bigger  Picture/Breaking  the  Pattern 


By  showing  the  underlying  structure  of  a  dynamic,  Causal  Loop  Diagrams 
show  where  best  to  apply  leverage  to  slow  or  stop  it — for  example: 

•  Change  negative  dynamics  into  positive  ones  by  running  them  backwards 

•  Slow  the  acceleration  of  unwanted  reinforcing  loops — “  When  you’re  in  a 
hole,  stop  digging” 

•  Change  the  limiting  value  a  balancing  loop  approaches  or  oscillates 
around  to  something  more  acceptable. 

Each  systems  archetype  has  specific  interventions  for  addressing  it 

Knowing  about  the  most  common  counter-productive  dynamics  is  the  best 
way  to  prevent  them 
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Acquisition  Archetype  Concept  Briefs 
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SEI  is  producing  a  set  of 
“Acquisition  Archetype" 
concept  briefs,  analyzing 
recurring  patterns  in 
actual  acquisition 
programs,  and 
recommending 
interventions  and 
preventative  actions 
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Next  Steps  and  Further  Information 


Extend  the  set  of  Acquisition  Archetypes 

•  Eleven  Acquisition  Archetypes  have  been  described  to  date 

•  Plan  to  identify  additional  acquisition  dynamics  and  root  causes 

For  additional  information 

•  Visit  the  SEI  website: 

—  http://www.sei.cmu.edu/proqrams/acquisition-support/pof-intro.html 

•  Upcoming  SEI  Technical  Note:  “ Archetypal  Patterns  of  Failure  in  the 
Acquisition  and  Development  of  Software-Intensive  Systems" 

•  Planned  2008  Workshop:  ‘Avoiding  Failure  in  Software  Acquisition ” 
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Software  Engineering  Institute 


Carnegie  Mellon 
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Federal  Information  Security 
Management  Act  (FISMA)  Operational 
Controls  and  Their  Relationship  to 

Process  Maturity 


Ronda  Henning 
rhenning@harris.com 


25  October  2007 


The  Basic  Premise  of  This  Presentation 


Proper  preparation  and  planning  makes  later 
phases  of  the  System  Development  Life  Cycle 

easier  to  conquer. 


NOTE:  FISMA  is  used  as  a  representative  standard.  Insert  the  security  guidance 
document  of  your  choice  in  the  context  of  this  presentation. 


Heim 
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About  FISMA 
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•  The  Federal  Information  System 
Management  Act  (FISMA) 

•  Consists  of  17distinct  families  of  security 
requirements 

•  Mandates  quarterly  vulnerability  reporting 
and  annual  progress  reports  to  GAO 

•  The  framework  for  how  to  report  is  left  to 
the  interpretation  of  the  parent  agency 

/ 
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FISMA  Control  Families 
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Management  Controls 

•  Risk  Assessment 

•  Planning 

•  System  and  Services  Acquisition 

•  Certification  &  Accreditation  (C&A) 


Technical  Controls 

•  Access  Control 

•  Audit  and  Accountability 

•  Identification  and  Authentication 

•  System  and  Communications 
Protection 


Operational  Controls 

•  Awareness  and  Training 

•  Configuration  Management 

•  Contingency  Planning 

•  Incident  Response 

•  Maintenance 

•  Media  Protection 

•  Physical  and  Environmental  Protection 

•  Personnel  Security 

•  System  and  Information  Integrity 


Controls  are  Complementary  and  rely  on  each  other  for  fulfillment 
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Relationship  among  controls 
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Operational  Controls 


•  People  Oriented 

-  Awareness  and 
Training 

-  Personnel  Security 

•  Physically  Oriented 

-  Environmental  Controls 

-  Media  Protection 

-  System  Integrity 

-  Contingency  Planning 
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Device  Oriented 

-  Configuration  Management 

•  Software 

•  Firmware 

•  Hardware 

-  Maintenance 

•  Routine 

•  Emergency 

-  Incident  Response 

•  What  is  an  incident? 

•  Reactive  v.  Proactive 
actions 

-  System  &  Information 
Integrity 

•  Is  the  data  corrupted? 

•  Is  the  system  image  valid? 

•  Are  they  current/accurate? 
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Device  Oriented  Requirements  h^\RR!S 


•  Harder  to  address  later  in  SDLC 

•  Frequently  neglected  in  development 

•  Reason: 

-  It’s  hard  enough  to  get  the  system  integrated 
and  working,  planning  for  later  operations  is 
left  to  the  student. 

•  In  reality: 

-  Planning  ahead  is  the  best  way  to  maintain  a 
proactive  assurance  posture 
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Security  Objective  of  Device  Controls  IJ^RRIS 


•  Define  and  maintain  a  known,  secure  state 

-  At  delivery  and  ongoing 

•  Systems  are  integrated  products 

-  Each  vendor  has  their  own  set  of  quality  and  security 
processes 

-  Monthly  patches,  quarterly  patches,  emergency 
patches 

-  Options  are: 

•  Working  system  with  vulnerabilities 

•  Semi-functioning  system  without  testing 

•  Cross  your  fingers  and  hope! 

-  Everything  works  with  the  patch  and  no  testing 

-  Nobody  tries  to  exploit  the  problems  before  you  fix  them 
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In  the  Ideal  World 


HAnms 


•  Regression  testing 

•  Impact  assessment 


In  reality: 

N  vendors  per  system 
Different  release  schedules 
N  software  applications 
System  may  have  over  1 00  products. 


Add  to 
Baseline 
Configuration 

•  Update  DM 

•  New  standard 
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Process  Integration:  A  Better  Way  tjAftRtS 


•  CMMI  processes  already  include  configuration 
management  and  change  management 

•  What  they  may  not  include  is  specific  processes 
associated  with  security  change  management 

•  Risk  must  be  addressed  in  the  process 


Likelihood 


Threat 

RISK  0 

Vulnerability 


1L 


Impact 


ifcita 
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Supplemental  Guidance 


HAnms 


•  System  Security  Engineering  CMM 

-  Add  security  relevant  functions  to  standard 
CMM  I  activities 

-  Incorporation  in  an  organization’s  standard 
process  framework  is  an  incremental  change 

•  A  Caveat: 


An  incremental  change  that  involves  careful 
component  management  ^ 

Accounting  at  a  more  granular  level 

•  All  the  component  software  entities 

•  Protocols,  reference  standards,  etc. 
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Mapping  Goals,  KPAs  and  FISMA: 


FISMA  Control: 


Specifies  what  must  be  managed,  what  artifacts  should  be  produced  for 
the  system.  Control  defines  the  compliance  baseline. 


Maps  to 

CMMI  KPA 

CMMI  KPA: 

Basic  process  guidance  & 
structure 

Specific  Guidance  for 
Security  Engineering 


SSE-CMM  KPA: 

•  Manage  Configuration  of  Security 
Components. 

•  Assess  security  impact  of  change? 

•  Define  change  management 
process 

•  Assess  risk  associated  with 
change? 

•  Document  risk  decisions 
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Implications 

•  Augmentation  to  existing  process  means  higher 
probability  of  organizational  acceptance 

•  Does  not  imply  use  of  automated  techniques: 
although  they  are  easier  with  larger  systems  and 
global  deployments 

•  Areas  for  automation: 

-  Asset  inventory 

-  Baseline  configuration  tracking 

-  Vendor  notification  and  update  service 

-  Deployment  tracking 
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Further  Implications  fjARRMS 

•  Starting  process  management  at  authority 
to  operate  is  too  late. 

•  The  baseline  is  established  by  then. 

•  May  not  have  been  monitored  and 
upgraded  throughout  development. 

-  It’s  hard  to  develop  code  on  a  moving  target 

-  Vulnerabilities  may  be  inadvertently  used  as 
part  of  the  system  feature  set 

-  Compromises  need  to  be  documented 
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Basic  Flow  ^l/?/?/S 

•  FISMA  families  explain  what  has  to  be 
done  (tangible  product) 

•  CMMI  provides  the  contextual  framework 
for  inclusion  of  FISMA  families  in  an 
integrated  set  of  engineering  processes 

•  SSE-CMM  defines  specific  process 
guidance  that  helps  an  organization 
develop  the  product 


> 

Mr 
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In  Summary 


HAnms 


•  Exact  correspondence  will  vary: 

-  Some  organizations  won’t  address  all  goals. 

-  Compensating  management  controls  can  be  traded 
against  technical  controls 


Goal  is  to  define  repeatable  process: 

-  Certification  and  accreditation  required  every  3  years 


Ongoing  monitoring  requirements  on  an  annual  basis 

Simpler  to  accommodate  the  requirements  within 
existing  processes  a 

SSE-CMM  and  CMMI  provide  guidance  and  ^ ^ 

placeholders  that  can  facilitate  compliance 
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Conclusion 


HAnms 


Starting  from  a  secure  foundation  is  easier 
than  trying  to  shore  up  an  unsound  one. 

Framework  for  security  improvement  is 
already  there  -  but  not  applied. 

Process  maturity  dictates  that  we  learn 
from  our  experiences  and  evolve. 


ifcita 
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For  More  Information 


HAnms 


FISMA: 

-  www.csrc.nist.gov 


•  SSE-CMM: 

-www.issea.org 

•  CMMI: 

-  www.sei.cmu.edu 
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Committee’s  Recommended  Policy 

Changes  to  DoD  5000 
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Current  Events 


hnu-wra  Tnnai  nnt  miitth  a  t>  i  tmi  orcir 


•  FY2007  DoDI  5000.2  Update 

-  “Fact-of-life”  changes  -  existing  policy  memos 

-  No  significant  impact  to  our  efforts 

•  FY2007  NDAA  Section  231  Report 

-  Report  to  Congress  on  T&E  Policy  & 

Practices 

•  Focus  on  new/emerging  acquisition  approaches 

•  Policy  changes  TBD 
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Current  Events  (cont) 


hlUFVITH  THEM  EH  MllTTH  A  T>  I  TMt  MiiV 


•  DT&E  Defense  Science  Board 

-  Formed  to  look  at  DT&E  roles,  practices 

-  Focus  on  improving  readiness  for  IOT&E 

-  No  impact  on  our  efforts 

•  NDIA  SED  DT&E  Committee  Focus  on 
DoD  5000  Recommended  Policy  Changes 

-  These  efforts  provide  complementary  or 
alternative  view 


October  23,  2007 


3 


Current  Events  (cont) 


hlUFVITH  THEM  EH  MllTTH  A  T>  I  TMt  MiiV 


DRAFT  POLICY  WOULD  BRING  ‘OPERATIONAL 
FLAVOR’  TO  WEAPON  TESTING: 


. .  By  incorporating  more  realistic  tests  of  system  reliability 
throughout  the  development  process,  officials  hope  to 
reverse  a  recent  trend  where  fewer  and  fewer  platforms 
meet  operationally  suitable  and  effective  standards  . . 


Quote  from  Charles  McQueary,  the  military’s  director  of 
Operational  Testing  and  Evaluation  (DOT&E),  told  reporters 
Oct.  19.  (DefenseNews.com) 
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Current  Events  (cont) 


hlUFVITH  THEM  EH  MllTTH  A  T>  I  TMt  MiiV 


From  Yesterdays  SE  Conference 
Opening  Session,  October  23,  2007: 

-Dr  McQueary’s  chart  -  Actions  to  improve 
Suitability  includes  the  DT&E  Committee 
White  Paper 

-Dr  Finley  made  a  comment  about  strong 
DT&E  prior  to  milestone  C 
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pHk  hTHfXITH  THkl  !l  mi  IMH.TT11  Jfe  T>  I  TT^ClL  iXrY 

2007  DT&E  Committee  Focus 


•  Three  Focus  Teams: 

-  Earlier  contractor  and  tester  involvement 

-  Integrated  DT/OT  and  DT  operational 
relevance  (combined) 

-Suitability 

•  Recommend  policy  changes 

-  Input  to  FY2008  DoD  5000  update 
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Project  Timeline 


hinrviTH  Tnnai  nnt  miitth  a  t>  i  tmi  orcir 


2007 

2008 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

Establish  Teams  ^ 
Review  Team  Scopes  ▲ 

NDIA  SE  Conf  -  Status 
Working  Meetings 
Present  Findings 

Draft  White  Paper 
Final  White  Paper 


i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 

i 


A 


A 


A 
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■  JiTHFM.TH  THk!  ll  •  .IE  IMH.TT1Y  A  T>  l  Tf^ClL  iXrY 

Focus  Team  Expectations 

•  Teams  expected  to  collaborate  between 
DT&E  Committee  meetings  and  present 
findings  at  the  meetings 

-  Collaborate  via  e-mail,  telecon,  etc. 

•  Product  is  a  White  Paper  on  DT&E  policy 
change  recommendations 

-  Draft  outline  available 
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■  JiTHFM.TH  THk!  ll  •  .IE  MH.TT1Y  A  T>  l  Tf^ClL  iXrY 

Focus  Team  Expectations 

•  October  Meeting  (in  San  Diego) 

-  Initial  draft  white  paper  outline 

•  Workshop  (tentatively  January  23-24,  2008) 

-  Initial  draft  of  white  paper 

-  Recommendations  presented  to  stakeholders  for 
comment 

-  Recommendations  sent  out  to  NDIA  SED  members 
for  comment 

•  April  2008  Submittal 

-  Team  leads  incorporate  stakeholder  and  SED 
member  comments,  prepare  final  recommendations, 
and  submit  through  NDIA  SED 
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Earlier  Contractor  and  Tester 

Involvement 


hnu-wra  Tnnai  nnt  ntm*™  a  t>  i  tmi  orcir 


•  Identify  the  “T&E  Community”  to  include  both 
contractor  and  government  DT,  OT  and  Live  Fire 
(LF)  organizations 

•  The  T&E  Community  needs  to  work  in 
collaboration  with  the  Acquisition  Community  in 
the  earliest  phases  and  decision  points  of  the 
acquisition  process. 

-  Make  T&E  Community  participation  and  products 
mandatory  at  all  early  phases 

-  Involved  the  T&E  Community  as  early  as  the  Concept 
Definition  phase  of  a  program.  This  needs  to  be 
detailed  in  the  JCIDS  process. 
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Earlier  Contractor  and  Tester 

Involvement 


hTHWTl  THNlll  mi  MllTTH  A  TV  I  Tf'ClL  IMJY 


•  The  Test  &  Evaluation  Strategy  (TES)  is 
developed  too  late  in  the  acquisition  process. 

-  The  TES  should  be  mentioned  in  the  Acquisition 
Strategy  and  fully  developed  by  the  T&E  Community 
prior  to  the  system  RFP/SOW. 

•  Awareness  of  test  readiness  is  lacking  as  a 
program  matures. 

-  Develop  a  Test  Readiness  Level  hierarchy  similar  to 
technology  readiness  levels  and  implement  as  part  of 
Concept  Definition. 

•  Need  earlier  incorporation  of  the  TEMP  in  the 
acquisition  process. 

•  There  is  a  lack  of  visibility  between  the  various 
T&E  organizations  across  the  T&E  phases. 

October  23,  2007 


11 


Integrated  DT/OT  and  DT 
Operational  Relevance 


hlUFVITH  THEM  EH  MllTTH  A  T>  I  TMt  MiiV 


•  Current  5000.2  allows  for  a  wide  variety  of 
integrated  testing  strategies  -  it  needs  to  provide 
more  specific  direction 

•  Need  to  ensure  clearly  defined  terms  -  Joint 
DT/OT,  Integrated  DT/OT,  Combined  DT/OT, 
etc. 

-  There  are  current  definitions  but  not  necessarily 
totally  agreed  to  yet  across  agencies 

•  DAU  is  a  way  to  work  the  commonality  of  terms 

-  Some  of  the  biggest  issues  are  cultural  between  the 
different  stakeholders 
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Integrated  DT/OT  and  DT 
Operational  Relevance 


hlUFVITH  THEM  EH  MllTTH  A  T>  I  TMt  MiiV 


•  Several  items  from  this  team  are  very  similar  to 
Team  1 

-  Earlier  involvement  &  Funding  needed 

-  Need  to  understand  all  the  Stakeholders  (i.e.  the  T&E 
Communities  that  should  be  involved) 

•  Specific  TEMP  sections  for  integrated  T&E  need 
to  be  included  &  coordinated  sooner  with  agreed 
to  terminology,  timelines,  etc 

-  Need  PM  &  T&E  in-sync  on  integrated  T&E 

-  How  to  transition  from  sequential  events  to  a  more 
integrated/concurrent  T&E  process 
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Suitability 


hnu-wra  Tnnai  nnt  ntm*™  a  t>  i  tmi  orcir 


•  High  percentage  of  programs  deemed 
operationally  effective,  but  NOT  suitable 

-  Ensure  system  suitability  is  a  key  feature  of 
T&E  Program 

-  Requirements  flowdown  must  include  user 
definition  of  mission  failure  and  scoring 
criteria 

-  Identify  potential  operational  failure  modes 
and  their  mission  impact  early 
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Suitability 


hnu-wra  Tnnai  rnt  ntm*™  a  t>  i  tmi  orcir 


•  Make  T&E  more  operationally  relevant 

-Test  entire  system  in  operational  environment 
and  scenarios 

•  Reliability  as  part  of  the  T&E  Program 

-  Require  mandatory  reliability  growth  and 
assessment  program  in  Request  For  Proposal 

-  Health  of  Reliability  program  considered  in 
each  program/technical  review  with  award  fee 
criteria 
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Summary 


hnu-wra  Tnnai  nnt  ntm*™  a  t>  i  tmi  orcir 


•  Significant  effort  by  T&E  and  Acquisition 
Communities  to  provide  more  successful 
testing  across  a  weapon  systems  life  cycle 

-  FY2007  DoDI  5000.2  Update 

-  FY2007  NDAA  Section  231  Report 

-  DT&E  Defense  Science  Board 

-  NDIA  SED  DT&E  Committee  Focus 
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Agenda 


•  UML®  artefacts  for  SE,  OMG  SysML™ 

•  Engineering  Change  Management 

•  A  Standard  Approach  to  Change  Management  for 
SysML 

-  I  SO  AP233 


T rademark  Notice 

OMG  SysML  Overview  slides  are  trademarked  or  registered 

trademarks  of  the  Object  Management  Group,  Inc.  in  the  United 
States  and  other  countries. 
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UML  artefacts  for  SE, 
OMG  SysML 
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The  "U"  means  "Unified" 


•  I  n  the  beginning,  there  were  several  software 
engineering  diagramming  techniques 

-  largely  pretty  pictures  for  human  consumption 

•  Unified  Modeling  Language  (UML®) 

-  is  their  merger/standardization  in  the  Object 
Management  Group  (OMG™) 

-  includes  numerous  diagrams 

-  includes  rigorous  underlying  model  of  the  information 
contained  on  those  diagrams 

-  is  extensible,  can  tailor  UML  to  create  new  languages 
called  UML  Profiles 
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UML  in  Systems  Engineering 


•  Sonne  UML  diagrams  are  useful  outside  the  software 
engineering  community 

-  E.g.  State  machines  to  simulate  systems  behavior 

•  Organizations  created  methodologies  for  using  UML  in 
Systems  Engineering 

•  SE  community  desired  more  commonality  and  so  the  OMG 
Systems  Modeling  Language  (SysML)  standard  was  born 

-  Same  thing  happened  for  Systems  Architecture  and  thus  the  OMG 
Unified  Profile  for  DODAF/MODAF  (UPDM)  was  born 


All  Presentation  Material  Copyright  Eurostep  Group  AB 


What  is  SysML? 


•  SysML  is  really  two  things 

-  A  set  of  graphical  notations  for  modeling  systems 

-  A  formal  specification  of  the  information  content  the 
icons  on  the  diagrams  represent 

•  a  subset  UML  language  model  with  SE  extensions 


•  SysML  was  developed  in  collaboration  between 
I NCOSE,  OMG  and  I  SO 

-  SysML  is  a  key  step  towards  the  Model  Based  Systems 
Engineering  vision 


All  Presentation  Material  Copyright  Eurostep  Group  AB 


Structure 
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Requirements 
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Engineering 
Change  Management 
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Location 

Management 
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Item 
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I  tern  -  Owner 


Perfect  Frame 


Vlii 

ri  ■  ■  I 


OWNER 
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Structure  -  Basic 


All  Presentation  Material  Copyright  Eurostep  Group  AB 


Structure  -  View  Based 
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I  tern  -  I D 


Perfect  Frame 


123-123 
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Item  -  Multiple  I D 


Fabulous  Factory  Inc 


Perfect  Frame 
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I  tern  -  Version 


-eurostep 
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I  tern  -  Version  2 
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I  tem  -  Views 


Assembly 

Instructions, 


Weight  =  12kg  Height  =  1234mm  Drawings, 

Color  =  Red  Length  =  4321  mm  Specifications, 


-eurostep 
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Document 


All  Presentation  Material  Copyright  Eurostep  Group  AB 


Change  Management  -  Design 
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Change  Management  -  Design 


Rear  Wheel  1 


Rear  Wheel  2 


INPUT  \\  //  OUTPUT 

> 


Proposed  solution  is 
stored  with  Planned 
effectivity 


Front  Wheel 


Create  a  proposal  of 
the  solution,  e.g.  new 
version  of  Rear  Wheel. 


Engineering  Change  Proposal 
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Change  Management  -  Design 
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Freezing 


•  Freezing  is  divided  into  two 
parts 

-  Freezing  Structure 

-  Freezing  Definitions  (prop,  doc) 

•  Freezing  can  be  done  on 
individual  views 
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Baselining 


The  baseline  object  can  explicitly 
point  out  the  complete  structure 
contained  in  a  baseline 


•  Except  baselining  a  structure,  a 
baseline  can  contain  all  other 
business  objects 
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Freezing  a  Baseline 


•  The  content  of  a  baseline  can  be 
edited  but  the  history  of  it  is 
always  kept 

•  Baselines  can  be  frozen  to  ensure 
that  the  specified  information  set 
can  be  re-called  at  all  times.  A 
frozen  baseline  can  not  be  edited! 

•  Enables  work  on  'open'  structures 
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A  Standard  Approach  to 
Change  Management  for  SysML 
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Extended  Lifecycle  Scope 


Full  Process  View 
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AP233  is  a  neutral  SE  information  model 
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Database 
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SysML-AP233  Alignment 


•  I NCOSE  drove  much  AP233  and  SysML  standardization 

-  OMG  for  SysML 

-  I  SO  TC184  SC4  I  ndustrial  Data  for  AP233 

•  AP233  and  SysML  teams  worked  together  to  align  them 

•  Aims  include 

-  Align  SysML  and  AP233  models 

-  Provide  meta- model  mapping 

-  Provisions  for  an  independent  public  domain  SysML/ AP233  API 

-  Set-up  of  data-exchange  test-bed 
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SE  Tool  Plug-fest 


•  The  SE  Tool  Interoperability  Plug-Fest 

-  SysML,  AP233  and  CADM  testing  capability  from  NIST 
and  DoD's  Systems  and  Software  Engineering  office 

•  Aims  to  support  testing  of  SysML  XMI  and  AP233 
XML  files 

-  J  ust  getting  started 

-  http:  //syseng.  nist.  gov/se-  i  nterop/  pi  ugfest/ 
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AP233-PLCS  Alignment 


They  share  a 
common  core 

-eurostep 
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CM  Items  in  AP233 


•  I  n  AP233,  the  CM  I  tern  concept  is  represented  as 
"Product"  or  any  of  its  subclasses 


•  Specify  SysML  concepts  that  map  to  AP233  CM 
items 

•  I  implement  SysML/ AP233  software 

-  Convert  the  internal  SysML  data  into  A233  data 
maintaining  reference  to  SysML  data  file  itself 
•  AP233  allows  reference  to  any  type  of  data  file 
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AP233  Change  Management  Schema 
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Use  Change  Management  Tool 

•  I  n  a  tool  that  implements  Engineering  Change 
Management 

-  Import  AP233  data  into  Item,  Item  Version,  etc. 

-  Check-in  the  SysML  data  file  itself 

-  Create  link  between  SysML  data  file  and  related  Item 

•  Use  CM  Tool  to  manage  Work  Requests,  Change 
Proposal  and  Change  Order  as  describe  earlier 
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Example  Requirements  Diagram 


Copyright  ©  2006,2007  by  Object  Management  Group. 


Zooming  in  on  the  Requirements 
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Example  Block  Definition  Diagram 


(Blocks) 
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Example  Heat  Exchanger  Flow  Ports 
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SysML  Underlying  Schema  for  Port 
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I  nitial  SysML  Map  to  AP233  CM  I  tem 
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Conceptually  merge  AP233/SysML 


+requests 

1 _ ± _ _ _ 

Work_request  +associated_request 

+request_id :  String  ^ 

+version_id :  String 

+description  :  String  [0..1  ]  +assigned_work_request 

+purpose :  String  -1 


0..* 


+in_response_to 


0..* 

Work_order 

+name :  String 
+description :  String  [0..1] 

+riems  j,  i  ./■ _ 

affected _item_se  feet 


75T 


Product 


+id :  String 

+name :  String  [0..1] 

+description :  String  [0..1] 


75T 


Document 


SysML  Port  is  a  kind 
of  Affected  I  tem  for 
AP233  Change 
Management 


i 


Port 

> 

> 

isBehavior  :  Boolean  =  false 
isSenice:  Boolean  =  true 

All  Presentation  Material  Copyright  Eurostep  Group  AB 


Future  integration  approach 

•  ISO  AP233  is  modeled  using  the  ISO  EXPRESS 
information  modeling  language 

•  I  SO  EXPRESS  being  submitted  to  OMG  for 
standardization,  called  MEXICO  project 

•  Enables  OMG  Model  Driven  Architecture 
technologies  to  be  applied  to  AP233  CM  of  SysML 
-  Tight,  direct,  standardized  AP233/SysML  alignment 
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I  ssues  for  future  work 


•  Working  with  multiple  versions  in  SysML  tools 


•  More  work  required  on  other  SysML  diagrams 
(e.g.  Parametrics) 

•  Links  between  Items  on  diagrams  and  the  SysML 
diagrams  on  which  they  appear  in  CM  tools 


•  Feedback  into  SysML  tools  from  CM  tools 
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Conclusions 


•  I  SO  AP233  enables  Engineering  Change 
Management  of  significant  aspects  of  SysML  and 
other  UML- based  models 

-  Brings  more  rigour  to  SE  processes 

•  However,  there's  still  plenty  of  work  to  be  done 


•  Proof- of- concept  development  underway  using 
our  Share- A-space  product  as  collaboration  and 
change  management  tool  for  MagicDraw  SysML 
tool 
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AP233  References 


•  DODAF/AP233  project  site 

-  http://www.exff.org/ap233 

•  AP233  standards  team  site 

-  http://www.ap233.org 

•  Eurostep 

-  http://ap233.eurostep.com  (kickoff  Nov  07) 

-  http://www.eurostep.com 

-  http://www.share-a-space.com 
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Overview  of  Presentation 


•  Background 

•  Definition  of  Unified  Simulation 

•  Real-life  Example 

-  Virtual  Systems  Integration  Lab  for  U.S.  Army 

•  Hurdles  to  Unified  Simulation 

•  Conclusions 

•  Recommendations 
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Background 


•  The  U.S.  Department  of  Defense  (DoD)  relies  on  a  multitude  of 
fragmented  simulations  to  assist  in  engineering  new  systems.  The 
DoD  recognizes  the  need  for  unified  simulation  environments  to 
enhance  the  value  of  new  models  and  help  achieve  its  defense 
transformation  goals;  a  major  example  of  this  is  the  U.S.  Army's 
OneSAF  program. 

-  However,  no  plan  exists  to  leverage  the  thousands  of  simulation  models 
that  remain  idle  on  shelves. 

-  Localized  efforts  by  the  government  and  its  contractors  to  unify  such 
models  have  been  marginalized  by  a  number  of  technical  and  non¬ 
technical  hurdles,  some  of  which  are  not  obvious. 

•  Overall  Goal:  Field  the  best  systems  for  the  future  military  force  in  the 
shortest  time  using  the  fewest  resources. 
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Definition  of  Unified  Simulation 


•  Unified  simulation  is  an  ambitious  goal  for  Systems  Engineering  that 
will  be  reached  once  the  following  criteria  and  capabilities  are  satisfied 
and  delivered: 

-  Interoperability  standards  allow  any  compliant  simulation  method  to  be 
incorporated  (e.g.,  HLA,  OneSAF) 

-  All  standalone  simulation  models  can  be  integrated  as  pieces  of  a  bigger 
puzzle  (e.g.,  Matlab,  Simulink,  C++) 

-  A  global  simulation  picture  provides  the  ability  to  “zoom  in”  on  any  level 
of  detail  ranging  from  systems  to  sub-components 

-  System  design  feedback  gets  generated  that  accelerates  feasibility  testing 
of  hardware  and  software 
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Real-life  Example 


•  Virtual  System  Integration  Lab  (VSIL)  for  the  U.S.  Army  Tank 

Automotive  Research,  Development  &  Engineering  Center  (TARDEC) 

-  VSIL  is  a  simulation  suite  for  accelerating  systems  engineering 

-  Tests  prototype  designs  prior  to  committing  to  a  physical  prototype 
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VSIL  Objectives 


Enhance  next-generation  vehicle  design 
and  development 

Improve  efficiency  of  simulation 
development 

Perform  cost-benefit  analysis  on 
component  models  up  to  full  deployments 

Transform  development  process  so  that 
new  vehicle  designs  benefit  from  the 
development  of  all  previous  vehicles 


Future  Systems 
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Result:  Virtual  Systems  Editing 
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•  Through  the  Virtual  Systems  Editor  (ViSE),  VSIL  provides  an 
integrated  design,  development,  and  simulation  toolset  to  enable 
automated  component  trade-off  analysis  and  requirements  generation 
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Hurdles  to  Unified  Simulation 


•  The  VSIL  team  encountered  the  following  hurdles  during  its  joint 
simulation  efforts  with  TARDEC: 

-  The  availability  of  models 

-  The  usability  of  simulation  construction  tools 

-  The  creation  of  reference  architecture 

-  The  complexity  of  simulation  results 

-  The  automation  of  repetitive  integration  tasks 

-  The  verification  &  validation  of  component  models 
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Hurdles  Explained 

•  The  availability  of  models 

-  Credibility  of  M&S  is  tied  to  the  availability  and  fidelity  of 
component  models  of  interest  (e.g.,  Mobility,  Suspension) 

-  Populating  a  useful  model  library  from  scratch  is  a  lengthy  task 
that  requires  vast  domain  expertise 

•  The  creation  of  Reference  Architecture  (RA) 

-  RA  defines  interfaces  required  by  models  to  be  leveraged  into  a 
unified  simulation  (e.g.,  RA  for  vehicle  electronics  component) 

-  Creating  RA  is  an  exhausting  task.  A  mature  RA  requires  constant 
re-factoring  over  time. 
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Hurdles  Explained 

•  The  verification  &  validation  of  simulation  models 

-  True  validation  of  models  is  only  possible  by  using  real  data  taken 
from  the  component  or  system  being  modeled,  or  by  using  the 
most  high-fidelity  models  available 

-  Requires  definitions  for  “high-fidelity”  models  and  tiers  of  model 
fidelity 

•  The  complexity  of  simulation  and  results 

-  Need  better  analysis  tools  to  process  output  data  faster 

-  Takes  more  time  to  execute  simulations  -  the  cost  of  accuracy 
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Hurdles  Explained 

•  The  usability  of  simulation  construction  tools 

-  Impacts  the  efficiency  of  model  verification  &  validation 

-  User-friendly  tools  encourage  more  use,  reduces  anxiety,  and 
builds  confidence 

•  The  automation  of  repetitive  integration  &  analysis  tasks 

-  Automated  model  wrapping  for  common  formats  is  highly  desired 

-  Need  to  automate  the  formatting  and  analysis  of  output  data 
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Conclusions  for  Military 

Simulation 


•  Simulation-based  engineering  is  a  vital  but  expensive  enterprise. 

•  Unified  simulation  is  an  ambitious  goal  that  will  accelerate  innovation 
and  make  systems  engineering  more  viable  in  the  long  run. 

•  Govt,  leadership  will  help  overcome  the  hurdles  to  unifying  military 
systems  engineering  simulations. 

•  The  DoD  is  the  only  organization  that  can  truly  unify  systems 
engineering  simulations  for  military  use.  Relying  solely  on  industry 
and  non-profits  like  SISO  to  accomplish  the  task  will  not  achieve  this 
goal  in  the  long  term  without  Govt,  mandate. 
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Recommendations  to  Maximize 
Simulation  Reuse  across  DoD 

•  To  establish  a  unified  approach  to  maximize  simulation  reuse,  the  DoD 
needs  to  mandate  a  standard  response  from  industry.  The  DoD’s 
mandate  must  include  provisions  for  three  broad  areas: 

-  1 .  Model  Sufficiency 

•  Are  high-fidelity  models  available? 

•  Are  they  compliant  with  interoperability  standards? 

-  2.  Tool  Usability 

•  Need  tools  that  highly  automate  the  M&S  process 

•  Software  tools  must  be  easy  to  use,  easy  to  learn,  and  fast 

-  3.  Process  Adoption 

•  Need  usage  to  get  credibility  and  continuous  improvement 

•  Write  model  deliverables  into  contracts 

•  Make  model  repositories  easily  searchable 
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Recommendations  to  Maximize 
Simulation  Reuse  across  DoD 

•  Mandate  wider  deployments  of  existing  efforts.  The  adoption  of 
simulation-based  processes  and  toolsets  in  the  defense  space  will  gain 
the  most  traction  when  mandated  with  ongoing  efforts. 

-  For  example,  existing  programs  such  as  OneSAF  should  publish  their  plan 
how  they  will  interoperate  with  new  models.  The  next  evolution  of 
OneSAF  should  incorporate  higher  fidelity  simulations  of  FCS  models, 
which  may  already  exist. 

-  Since  OneSAF  is  expected  to  be  a  platform  for  other  services  if  it 
continues  to  be  successful,  this  should  trigger  a  number  of  action  items 
including:  discovering  needed  models,  identifying  interoperability 
protocols,  and  designing  necessary  extensions  to  incorporate  OneSAF  into 
new  programs. 
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Recommendations  to  Maximize 
Simulation  Reuse  across  DoD 

Employ  a  bottom-up  approach  to  unifying  simulations. 

-  Experience  shows  that  a  bottom-up  approach  to  unifying  simulations  is 
superior  to  a  top-down  approach. 

-  For  example,  the  expansive  JSIM  project  that  preceded  OneSAF  failed 
due  to  the  management  burdens  of  operating  as  a  joint-service  project. 

Account  for  ongoing  simulation  interoperability  efforts. 

-  A  unified  approach  relies  on  simulation  interoperability. 

-  Consider  how  ongoing  infrastructure  developments  in  the  DoD 
community  will  fit  in,  including  HLA,  BOMS,  SEDRIS,  and  MSDL. 

Populate  government-owned  model  repositories.  Let  industry  maintain 

proprietary  repositories  with  interface-based  model  access. 

-  Interface  with  decentralized  repositories  based  on  service  agreements 

-  Provide  real  data  for  Govt,  engineers  and  support  contractors  to  use 
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Recommendations  to  Maximize 
Simulation  Reuse  across  DoD 

Establish  a  validation  program  for  simulation  models. 

-  A  program  is  necessary  to  verify  the  adequacy  of  simulation  models. 

-  Can  be  run  by  a  university  center,  similar  to  the  way  Johns  Hopkins  was 
contracted  to  perform  HLA  RTI  compliance  testing. 

Invest  in  a  standard  simulation  design  environment. 

-  Investing  in  a  standard  simulation  design  environment  will  enable  the 
DoD  to  send  a  tangible  mandate  to  its  PEOs  and  contractors. 

-  Identify  a  software  toolset  that  is  easy  to  use,  accurate,  useful,  &  flexible. 

Require  the  delivery  of  component  models  developed  under  contract. 

-  Govt.  &  Industry  need  standardized  tools  to  handoff  and  evaluate  models 

-  The  DoD  needs  more  automated  M&S  capabilities  and  should  buy  better 
tools  to  effectively  manage  M&S. 
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Question  &  Answer 

•  Please  address  questions  after  the  conference  to  Kevin  Tang: 

ktang@  Cybernet . com 
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Lessons  Learned  in  Seamless 
Integration  of  CMMI,  TSP,  and  PSP 

Why  All  Three  Are  Needed 


1  Oth  Annual  Systems  Engineering 

Conference 

October  24,  2007 
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>  Issues 

♦  Quality  and  Schedule 

♦  Rational  Management  and  Commitment 

♦  Insanity  and  Malpractice 

>  Three  Improvement  Perspectives 

♦  Organization  -  CMM/CMMI 

♦  Individual  -  PSP 

♦  Team-TSP 

>  Seamless  Integration  of  CMMI,  PSP,  TSP 

♦  The  glue  -  Process  Improvement  Proposal 

♦  AIS  Experience 

>  Lessons  Learned 
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Quality  Is  More  Important  Than 

Schedule 

“In  today’s  software  marketplace,  the 
principal  focus  is  on  cost,  schedule,  and 
function;  quality  is  lost  in  the  noise.  This  is 
unfortunate  since  poor  quality  performance 
is  the  root  cause  of  most  software  cost  and 
schedule  problems.” 


Watts  Humphrey 
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Rational  Management  -  Developers 


>  When  pressed  for  early  deliveries,  the 
responsible  team  members  say 

“I  understand  your  requirements,  I  will  do  my 

utmost  to  meet  it,  but  until  I  make  a  plan,  I  can 
not  responsibly  commit  to  a  date” 
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Rational  Management  -  Managers 


>  When  pressed  for  early  deliveries,  the 
responsible  managers  say 

“I  trust  you  to  create  an  aggressive  and  realistic 
plan,  I  will  review  the  plan,  but  I  will  not 
commit  you  to  a  date  that  you  can  not  meet” 
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Rational  Management  -  Principles 

>  Set  challenging  goals 

>  Get  the  facts 

>  Use  facts  and  data 

>  Anticipate  and  address  problems 
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Insanity  or  Malpractice? 

Insanity 

Doing  the  same  thing  over  and  over  and 
expecting  a  different  result 

Malpractice 

An  organization  which  does  not  have  a 
top-management- sponsored 
continuous  improvement  initiative  in  place 
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Organization  Improvement 


Capability  Maturity  Model 


Level 

Focus 

Key  Process  Areas  (KPA) 

5  Optimizing 

Continuous  process 
improvement 

Defect  prevention 

Technology  change  management 
Process  change  management 

4  Managed 

Product  and  process 
quality 

Quantitative  process  management 
Software  quality  management 

3  Defined 

Engineering  process 

Organization  process  focus 
Organization  process  definition 
Training  program 

Integrated  software  management 
Software  product  engineering 
Intergroup  coordination 

Peer  reviews 

2  Repeatable 

■ 

Project  management 

Requirements  management 

Software  project  planning 

Software  project  tracking 

Software  quality  assurance 

Software  configuration  management 
Software  subcontract  management 
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Comparing  SW-CMM  to  CMMI 


Level  5 
Optimizing 


SW-CMM  key  process  areas  CMMI  Process  Areas 

Defect  Prevention  ►  Causal  Analysis  and  Resolution 

Technology  Change  Management— ►Organizational  Innovation  and  Deployment 
Process  Change  Management 


Level  4 
Managed 


Quantitative  Process  ManagemenfyjDrganizational  Process  Performance 
Software  Quality  Management  ►quantitative  Project  Management 


Level  3 
Defined 


Organization  Process  Focus 
Organization  Process  Definition 
Training  Program 
Integrated  Software  Management 


Software  Product  Engineerin 


Intergroup  Coordination 
Peer  Reviews 


Organizational  Process  Focus 
Organizational  Process  Definition 
Organizational  Training 
Integrated  Project  Management 
Risk  Management 
Requirements  Development 
Technical  Solution 
Product  Integration 
Verification 
Validation 

Decision  Analysis  and  Resolution 


Level  2 
Repeatable 


Requirements  Mgmt  Requirements  Management 

Software  Project  Planning  Project  Planning 

Software  Project  Tracking  &  Oversight  Project  Monitoring  and  Control 
Software  Subcontractor  Management  Supplier  Agreement  Management 
Software  Quality  Assurance  Product  &  Process  Quality  Assurance 

Software  Configuration  Management  Configuration  Management 

Measurement  and  Analysis 
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Issues  Addressed  by  CMM 


>  Getting  management  attention 

>  Maintaining  long-term  improvement  focus 

>  Guiding  the  improvement  work 
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CMM  Results  -  Schedule 

GM 

>  Average  number  of  days  late  in  meeting  milestones  declined  from  over 
50  days  to  fewer  than  10  following  organization  focus  on  CMMI 


General  Motors  Presentation,  SEPG,  Boston,  MA,  2003 
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CMM  Results  -  Defects 


■  The  TSP  in  Practice,  SEI  Technical  Report,  September  2003 
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CMM  Problems 

>  No  simple  model  could  precisely  measure  process 
maturity  and  complex  models  are  not  useful  in 
guiding  improvement 

>  CMM  consciously  focused  on  what  organization 
should  do,  not  on  how  they  should  do  it 

>  The  teamwork  practices  and  personal  disciplines 
required  for  quality  software  work  are  almost 
entirely  issues  of  how,  and  not  just  what 

>  Because  engineers  will  not  change  the  way  they 
work  without  very  specific  guidance,  the  CMM 
does  not  change  engineering  behavior 
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The  Real  Need 


>  The  need  is  not  for  lots  of  process  data  but  for 
engineers  who  gather  and  use  that  data 

>  What  would  happen  if  software  professionals  used 
sound  engineering  practices? 

-  made  and  followed  detailed  plans 

-  gathered  and  used  historical  data 

-  measured  and  managed  quality 

-  analyzed  and  improved  their  processes 

>  The  need  is  for  a  Level  5  Process  at  the  individual 
level 
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Self  Improvement 
From  Project  To  Project 

“Y ou  can  not  stand  still,  so  you  should  treat 
every  project  as  a  way  to  build  talent  rather 
than  merely  treating  your  talent  as  a  way  to 
build  projects” 


Watts  Humphrey 
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Self  Improvement 
Personal  Software  Process  -  1 


r 


PSPO 

Current  process 
Time  recording 
Defect  recording 
Defect  type  standard 


PSP0.1 

Coding  standard 
Size  measurement 
Process  improvement 
proposal  (PIP) 


establishing  a  measured  performance 
baseline 
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Source:  Software  Engineering  Institute 
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Self  Improvement 
Personal  Software  Process  -2 

>  At  the  end  of  the  PSP  training,  developers 
know  how  to: 

♦  Consistently  gather  size,  time,  and  defect  data 

♦  Make  commitments  based  on  historical  data 

♦  Analyze  personal  data  to  answer  questions 

-  Where  am  I  spending  my  time? 

-  What  are  my  common  defects? 

-  Where  do  I  inject  the  defects? 

-  What  goals  do  I  need  to  set  to  improve? 
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%  Deviation 


PSP  Results  -  Schedule 

AIS 


Schedule  Deviation  Individual  Value  Control  Chart  - 

Commercial  Systems 

350 
300 
250 
200 
150 
100 
50 
0 
-50 
-100 
-150 

Date  of  Project  Phase  Start 

O  Individual  Data  Points  - Mean  —  -  —Upper  Natural  Process  Limit  —  -  —Low er  Natural  Process  Limit  -  -  -  -  One  Standard  Deviation 
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PSP  Results  -  Defects 

AIS 
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PSP  Problems 


>  To  do  quality  work,  engineers  need  a  detailed  plan 
and  a  defined  process 

>  Without  the  process,  they  cannot  make  detailed 
plans,  take  consistent  measurements,  or  track  their 
work  against  the  plan 

>  However,  when  engineers  have  a  project  to 
deliver,  they  are  rarely  willing  to  take  the  time  to 
define  a  complex  process,  even  when  they  know 
how 

ais _ 
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The  Real  Need 


>  Need  a  mechanism  to  guide  teams  through 
defining  their  processes  and  making 
complete,  precise,  and  detailed  plans 

>  Need  a  vehicle  to  help  organizations 
capitalize  on  the  potential  benefits  of 
disciplined  teamwork 
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Team  Improvement 
Jelled  Teams 


“The  speed  with  which  organizations  form 
and  deploy  teams  is  the  single  most 
important  factor  in  determining  their 
competitive  success” 

“Jelled  teams  are  the  most  powerful  tool  ever 
devised  for  doing  challenging  work” 

>  Watts  Humphrey 
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Team  Improvement 
Self-directed  Teams 

>  Characteristics  of  self-directed  teams 

-  Sense  of  membership  and  belonging 

-  Commitment  to  a  common  team  goal 

-  Ownership  of  the  process  and  plan 

-  The  skill  to  make  a  plan,  the  conviction  to 
defend  it,  and  the  discipline  to  follow  it 

-  Dedication  to  excellence 
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Building  Self-directed  Teams 
The  TSP  Launch  Process 

Day  1  Day  2  Day  3  Day  4 
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Self-directed  Teams 
Project  Tracking  Issues  -  1 

>  With  PSP  training,  developers  know  how  to  plan, 

schedule,  and  track  their  work 

>  TSP  teams  use  these  PSP-learned  methods  to  make 

detailed  plans 

-  Tasks  are  no  more  than  10  task  hours  each 

-  Task  time  is  recorded  daily 

-  EV  is  measured  weekly 

>  You  can  tell  project  status  to  within  10  task  hours 

>  TSP  teams  regularly  report  their  status 
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Self-directed  Teams 
Project  Tracking  Issues  -  2 

>  Project  schedules  slip  a  day  at  a  time 

>  If  you  cannot  precisely  measure  project  status,  you 

will  not  know  where  projects  stand 

>  Without  such  knowledge,  you  cannot  address 

schedule  problems  in  time  to  fix  them 

>  With  the  TSP,  you  can 

-  closely  monitor  team  performance 

-  address  problems  in  time 

-  consistently  meet  schedules 
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Hours 


TSP  Results  -  Task  Hours 


oor^i-^-cocMcoooor^o^t- 


Week 


Source:  Allied  Signal 
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3/10 


Defects/ KLOC 


TSP  Results  -  Defects 


Defect  Density  of  Delivered  Software 

7  R 


CMM  CMM  CMM  CMM  CMM  TSP 
Level  1  Level  2  Level  3  Level  4  Level  5 


Ref:  SEI  Technical  Report  2003-014 
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Transforming  the  Culture 


Organization 


Project 


Individual 


Level  3 

-Organization  develops 
standard  processes 

Common  culture 


Level  5 

Proactive 
■n  p  no  veme  nts 
Agile  culture 


Level  2 

Project  mg  ns.  establish 
discipline  &  stability 

Commitment  culture 


Level  4 

Development  process 
managed  statistically 

Precision  culture 


T  rust 


Level  1 

''  ^  1 

Level  S 

Ad  Hoc  processes. 

Opportunistic 

inconsistent  results 

improvements 

Hero-driven  culture 

Empowered  culture 

Borland 
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Source:  “From  MCC  to  CMM”,  Dr.  Bill  Curtis,  DC  SPIN,  April  2006 
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Process  Improvement  Principles 

>  It  takes  time,  skill,  and  money  to  improve  the 
software  process 

>  To  improve  the  software  process,  someone  must 
work  on  it 

>  Unplanned  process  improvement  is  wishful 
thinking 

>  Automation  of  a  poorly  defined  process  will 
produce  poorly  defined  results 

>  Improvements  should  be  made  in  small  steps 

>  Train,  train,  train! 


Source:  Managing  the  Software  Process,  Watts  Humphrey 
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Empowered  Culture 
Process  Improvement  Proposals  (PIPS) 


PROCESS  IMPROVEMENT  PROPOSAL  (PIP) 


PIP#  : 
Written  By: 


Process  Name  : 


Improvement  Description : 

F 


Authors) :  r  j 


Improvement  Benefits  (Check  One)  : 

C  Document  Improvement  C  Reduced  Cycle  Time 
P  Improved  Quality  C  Reduced  Risk 


Project :  r  j 


Key  Process  Area : 


Benefits  Description  (Quantify  Where  Possible)  : 

(Attach  files  if  needed) 

F 

s 

Attach  the  PIP  Pilot  Report  here  (if  applicable):  r  j 


▼  SEFG  Evaluation 
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Submit 
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The  AIS  PIP  Process 


Process  Change  Management 

Leveraging 


Create  New  PIP 


Open  PIP  Queue 


Feedback  / 


About  PIP  Database 


Initiating)  pip 


\lmplement\ 

a  \  Acting 

Integrate  | 


DiagnosingX 


pip 

\  Meeting 


/  Assign 
/  Resources 


Establishing 
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AIS  PIPs  Summary 


Jan  22,  1992  -  To  date 


No.  of  PIPs  submitted  1502 

No.  of  PIPs  implemented:  972 

No.  of  PIPs  by  improvement  category: 

•  Improved  quality  232 

•  Reduced  cycle  time  86 

•  Reduced  risk  63 

•  Improved  documentation  161 

•  Not  categorized  410 
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Sample  PIPs  -  Organization  Process 


>  Incorporate  the  TSP  into  the  AIS  CPIW  as 
suggested  by  the  attached  work  products 
(ProjectCommitmentProcess.zip)  which  reflect  the 
current  practice 

>  Change  Launch  meeting  9A  so  that  review  is  held, 
not  only  by  management,  but  also  peer  Project 
Managers.  Accordingly,  these  same  individuals 
may  need  to  be  present  in  meeting  1 B 
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Sample  PIPs  -  Team  Process 

>  For  UI  component  enhancements,  change  process 
to  do  Design  Inspection,  Test  Case  Inspections 
and  Code  Inspections  after  Compile 

>  For  components  where  performance  requirement 
is  critical,  execute  two  rounds  of  unit  test 

-  Unit  test  of  performance  test  cases  before  code 
inspection 

-  Unit  test  of  features  after  code  inspection 
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Sample  PIPs  -  Personal  Process 


>  Reduce  phase  distribution  %  for  Design 
Review  for  UI  Components 

>  Update  Personal  Review  Checklist 

>  Batch  process  E  Mail  three  times  a  day 

>  Move  end  of  day  post  mortem  to  start  of 
day  to  process  and  analyze  previous  day’s 
data 
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Lessons  Learned  -  1 


>  While  models  are  useful  to  indicate  where 
improvements  are  needed,  only  committed  people 
can  make  the  improvements 

>  A  supportive  management  environment  that 
rewards  disciplined  behavior  is  absolutely 
essential 

>  Timely  feedback  on  the  status  and  disposition  of 
the  PIPs  is  important  to  sustain  the  PIP 
mechanism  and  feeling  of  empowerment 

>  Do  not  need  to  wait  till  level  5  to  start 
implementing  process  change  management 
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Lessons  Learned  -  2 


>  While  CMM  is  necessary  as  an  organizational  capability 
improvement  model,  it  is  not  sufficient  to  change 
engineering  behavior;  the  PSP  provides  the  detailed  “how 
to”  for  improvement  at  the  individual  level 

>  The  TSP  provides  the  management  framework  for 
continuously  improving  self  directed  teams.  The  PIP 
mechanism  is  key  for  team  ownership  of  the  project’s 
process  and  commitment  to  improve 

>  CMM,  TSP,  and  PSP  all  three  are  needed  for  an 
integrated  approach  to  model  based  improvement  at  the 
organization,  team,  and  individual  levels  without  the  risk 
of  sub-optimization 
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Trademarks  and  Service  Marks 

>  The  following  are  service  marks  of  Carnegie  Mellon  University. 

♦  CMMISM 

♦  Team  Software  ProcessSM 

♦  TSPSM 

♦  Personal  Software  ProcessSM 

♦  PSPSM 

>  The  following  are  registered  trademarks  of  Carnegie  Mellon 
University. 

♦  Capability  Maturity  Model® 

♦  CMM® 

♦  Capability  Maturity  Model®  Integration 

♦  CMMI® 

♦  CERT® 
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Contact  Information 


Girish  Seshagiri 

Advanced  Information  Services  Inc. 

(703)  286  0781 
Email:  girishs@advinfo.net 
Website:  www.advinfo.net 
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Aging  Aircraft  Sustainment 
with  Non-Standard  Engineering 

NDIA  10th  Annual  Systems  Engineering  Conference 
Hyatt  Regency  Mission  Bay,  San  Diego,  CA 
October  22-25,  2007 


Kendal  Hinton 
404-407-6042 

kendal.hinton@gtri.gatech.edu 


Chris  Fowler 
404-407-7094 

chris.fowler@gtri.gatech.edu 


Evolution  of  Avionics  Systems 


FROM... 

•  Single-Function,  stand-alone 
characterized  by  multiple 
subsystems 

•  Connected  multiple  analog 
signals  using  point-to-point 
wiring,  to  provide  a  single 
function 


TO... 

•  Digital  technology  for 
information  transfer 

•  Allowed  network  sharing  of 
the  physical  interface 

•  Reduced  number  of 
interconnections  within  the 
airframe 


MIL-STD-1 553 


•  Result  of  a  cooperative  effort  between  the  military 
and  industry 

•  Defines  the  electrical  and  protocol  characteristics 
for  a  digital,  serial  communication  standard  among 
systems 

•  From  its  initial  release  in  1973,  the  standard  has 
been  revised  and  updated  to  reflect  lessons  learned 
from  implementation. 

•  Currently  standard  version  is  revision  B,  Notice  6 


MIL-STD-1553B  Notice  6 


•  Defines  the  data  bus  network  as  a  a  main  bus  cable 
to  which  stubs  are  attached  and  terminals  are 
connected  to  the  stubs 

•  Voltage  waveforms  arrive  at  different  terminals  with 
the  least  amount  of  distortion 

•  Major  parameters  affecting  waveform  quality  are 
bus  length,  number  of  stubs,  and  locations  and 
lengths  of  stubs 


A  Design-to-Standard  Bus 


2-Stub  ftus  Coup'^i 


]-£lirfs  Bls  Coupler 


http://www.n-diqital.co.ip/Milestek/diaqramandtechinf/Mil1553bComp.intro.files/SVS.JPG 


Non-Standard  A/C  1553  Wiring 

Analysis 


LEGACY  ISSUES... 

•  While  strides  are  being 
made  to  integrate  avionics 
systems,  the  physical 
infrastructures  on  the  target 
platforms  may  not  be  up  to 
the  bus  standard. 

•  Installing  wiring  that 
conforms  to  the  standard  on 
any  legacy  system  can  be 
costly 


POSSIBLE  SOLUTION... 

•  Using  non-compliant  wiring 
installed  on  an  aircraft,  can 
systems  reliably  exchange 
information  over  the  bus? 

•  Beneficial  to  derive  and 
implement  an  analysis 
process 


Non-Standard  A/C  1553  Wiring 

Analysis 


To  ensure... 


•  Performance 

•  Maintenance 

•  Supportability 


Plan  to... 

•  Develop  Spice  Models 

•  Execute  Lab  Tests 

•  Perform  SPICE  Analysis  of 
Actual  A/C  Wiring 

•  Perform  Lab  Analysis  of 
Actual  A/C  Wiring 


Existing  A/C  1553  Wiring 

F-16C+  Block  Diagram 


Georgia 

Tech 


II  I  I 


Examining  Signal  Quality  on  the  Bus 

Network 

•  GOAL  -  To  transfer  voltage  waveforms  with  minimum 
distortion 

•  To  determine  whether  or  not  a  network  will  perform  reliably, 
its  characteristics  are  measured  and  compared  to  the 
requirements  of  the  standard. 

•  The  quality  of  the  waveform  is  determined  by  examining  it  in 
the  following  respects: 

•  Amplitude 

•  Zero-crossing  distortion 

•  Waveform  tailoff 


Test  Waveform 


Laboratory  Mockup 


Coupler 


10  '  Twinax 


'  Twinax 


T_ 


40  '  Twinax 


PC  with  PASS 
As  BC 


39  '  Coax 


184 

Circuit 

1 


Scope 


PC  with  PASS 
As  RT 


Transformer  Circuit  Solution 


Georgia 


Existing  A/C  1553  Wiring 


EW  DATA  BUS  Terminator 


EWDAT+ 


EWDAT- 


Computer  Simulation 


•  Computer  Simulation  provides  an  approximation  of 
the  quality  of  the  signal  that  can  be  achieved  with  a 
hardware  mockup 

•  A  SPICE  program  was  used  to  model  a  transmission 
line  defined  by  the  characteristics  of  the  standard 
and  non-standard  wiring 

•  The  transmission  line  was  linked  to  other 
components,  i.e.  resistors  and  transformers,  to  form 
the  standard  1553  bus  design 


SPICE  Bus  Configuration 


Impact  of  Non-Standard  Wiring 


BC  commands  one  word 
transmit  from  RT  (0x0C21  1-T-1- 
1) 

RT  answers  with  status  word 
followed  by  1  data  word 

Examine  waveform  quality 
(MIL-HDBK-1 553,  §40.9) 

•  Amplitude 

•  Zero-crossing  distortion 

•  Tailoff 


Georgia 

Tech 


Input  Waveform  Amplitude  at  RT 


40’  Twinax 

•  Measured  Voltage 

•  Twinax:  5.4  v 

•  Coax:  3.12  v 

•  Requirement:  0.86-  14.0  v 

39’  Coax 


Input  Waveform  Zero-Crossing  at  RT 


Measurement  shown  is  zero¬ 
crossing  for  first  bit  of 
command  word  to  the  first 
bit  of  the  data  word 

Measured  Time 

•  Twinax:  2.02  |js 

•  Coax:  2.04  |js 

Requirement:  2  us  ±150  ns 


Input  Waveform  Tailoff  at  RT 


•  Voltage  must  be  less  than 
±250  mV  for  the  period 
beginning  2.5  ps  following 
the  last  mid-bit  zero¬ 
crossing. 

•  Both  waveforms  exhibit 
clear  end  to  data  waveform. 


Impact  of  Non-Standard  Wiring  -  BC 


Stop 

_ 

[  Q 

1 

Q 

M  10. Ops  A  Chi  f  1 00 mV 

14  Aug  2006 

ii->  27.0000ms 

13:27:04 

BC  commands  1-word 
transmit  from  RT  1  (0x0C21 
1  -T-1  -1 ) 

RT  1  answers  with  status 
word  followed  by  1  data 
word 

Examine  waveform  quality 
(MIL-HDBK-1 553,  §40.9) 

«  Amplitude 

•  Zero-crossing  distortion 

•  Tailoff 


Input  Waveform  Amplitude  at  BC 


•  Measured  Voltage 

•  Twinax:  5.28  v 

•  Coax:  2.82  v 

•  Requirement:  0.86-  14.0  v 


Input  Waveform  Zero-Crossing  at  BC 


Measurement  shown  is  zero¬ 
crossing  for  first  bit  of 
command  word  to  the  first 
bit  of  the  data  word 

Measured  Time 

•  Twinax:  2.0  jjs 

•  Coax:  2.06  |js 

Requirement:  2  us  ±150  ns 


Input  Waveform  Tailoff  at  BC 


•  Voltage  must  be  less  than 
±250  mV  for  the  period 
beginning  2.5  ps  following 
the  last  mid-bit  zero¬ 
crossing. 

•  Both  waveforms  exhibit 
clear  end  to  data  waveform. 


Impact  of  High  Traffic  Level 


PreVu 


M  1 0OjiS  A  Chi  "jt  200mv 

31  Aug  2006 

II »-»-  -21.6000|JS _ 15:40:32 


•  Maximum  bus  loading  was 
added  to  the  analysis 

•  Message  changed  to  a  32- 
word  transfer  at  the 
minimum  inter-message 
gap,  resulting  in  a  bus 
loading  at  just  over  99% 


Georgia 

Tech 


Impact  of  High  Traffic  Level 

Input  Waveform  Amplitude 


PreVu 

M  200JJS 

▼ 

u 

± 

:  j  :  :  : 

:  j  :  :  : 

;  3h  ;  :  ; 

:  i  :  :  : 

...  | . I . 

Z  4.00ms  A  Chi  I 

200mV 

31  Aug  2006 

II  >▼  -402.560MS 

15:45:45 

•  Measured  Voltage 

•  5.0  v 


Impact  of  High  Traffic  Level 

Zero-crossing  Deviation 


PreVu 

M  200JJS 

U 

± 

:  :  :  :  j 

:  :  :  :  i 

;  ;  ;  ;  1 

:  :  :  :  i 

. i" 

Z  I.OOjiS  a'  Chi  J"  200mv 

31  Aug  2006 

gj-r  -4 14. 540 us  15:49:51 

•  Measured  Time 


Impact  of  High  Traffic  Level 

Input  Waveform  Tailoff 


•  The  waveform  exhibits  a 
clear  end  to  data  waveform 
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Use  of  non-standard  wiring  OK? 


•  Short  answer:  Yes. 

•  What  gets  “done-to”  should  be  “un-done”  at  the 
terminal  end. 


Non-Standard  A/C  1553  Wiring 

Analysis 


•  Sufficient  Performance 

•  Low  Maintenance 

•  Easy  Supportability 

•  Minimal  Cost 
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Architecture  versus  M&S? 
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System 

Architecture 

(DODAF) 


I 


System 
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System 

Ops  Concept  Architecture 

(DODAF) 


System 


Bridge  the  Gap 

Architecture  and  M&S 


Simulation  / 

Evaluation 

Concordance 


Introduction 

Research  Objectives  /  Implications 
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Demonstrate  an  improved  process  of  using 
architectures  to  evaluate/refine  a  proposed  system  concept 


Application: 

Weapon  Borne  Battle  Damage  Assessment  (WBBDA) 

System  Concept  (2015-2025  time  frame) 

■  Develop  DODAF  system  architectures  (both  “as-is”  and  “to-be”) 

■  Key  Products:  OV-1,  OV-2  (nodes),  OV-5  (activities),  OV-6a  (rules), 
OV-6b  (state  transition  diagram,  or  discrete  event  sim),  OV-7  (data) 

■  Develop  evaluation  models  directly  from  the  system  architectures 

■  Analyze  results  to  identify  key  design  parameters  that  can  translate  to 
system  requirements  and  Key  Performance  Parameters  in  the  JCIDS 


Methodology 
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■  Develop  Architecture  based  on  joint  ops  concept 

■  DoDAF  architecture  views 

■  Compare  AS-IS  and  TO-BE  architectures 

■  Develop  and  use  simulations  based  on  architecture 

■  Analytical  Model  -  Excel,  with  Decision  Analysis  add-in 

■  Discrete  Event  Simulation-  Rockwell  Arena 

■  Evaluate  the  system  concept  based  on  the  results 
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Current  BDA  Ops  Concept 

ov-i 
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Architecture 

AS-IS  OV-2  Operational  Node  Connectivity 
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The  BDA  Cycle 
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So  what  is  WBBDA  ? 

“To-Be"  OV-1 


Global  r/tf  orndfood  Gnd 


Satellites 
Collect  BDA 


Attac  k  Assets 


Oft  B&ard  Sensors 

Collect  BDA 


on  B&ard  senior? 

,  Collect  BDA  J 


Stic  Lira  i 
Target? 


Weapon  Bonw 

Battle  Damage  Assessment  \ 


Subsurface 

Target 


Command  and  Oorrttal 
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Architecture 

TO-BE  OV-2  Operational  Nodes  Diagram 
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The  WBBDA  enabled  BDA  Cycle 


10 


OV-5  Activity  Diagram 
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rifanrj.-jon  hetwnrir 
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At  ESD  Level... 


Major 

CCJO 

Combat 

Operations 

...and  System  Level 

Perform 

BDA 


Architecture 

OV-6a  Rules  Model 


Target 


Net-Centric  Erwt  onnwnt  JFC 


Attach  Effectij 

r*s^ 

Reportng 


M 

Data  _ 


Re-attack  feMe  -lendatKm 
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Pt«-Str*e  Target  Data 


Datl  Sw«j  Target  am  # 

'  m 


DiA  8D-A  Reference  Gmaejnes 


Target  ReAttsdt  Tasking 


?, !  SREP 


WBBDA 

Weapon  Some  Sensors| 

Onboard  Sensors 
Off  Board  Sensors 
Assesameni  E  nities ' 

TBUCS 
C  l  Briefings 

Information  Network 


WBBDA 
Automated 
BDA  Results 


»A  Results  i 

/ 

t  Reattack  Effects 

Jt  7 


Perform 

ReAttack 
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Architecture 

Method  for  Metrics 
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■  MOEs  Established  in  ICD 


Measure  of  Effectiveness 

Numerator 

Denominator 

1.  AOR  Coverage  (AORC)  -  %  of  targets  that  receive  BDA  results 

#  targets  BDA  is 
collected  on 

#  of  targets 
attacked  per 
package 

2.  Total  Time-Obscured  Target  (TT-OT)-Looks  at  total  time  from  the  completion 
of  the  attack  strike  (on  obscured  targets)  to  the  point  when  all  BDA  assessment 
and  dissemination  is  complete. 

time 

n/a 

3.  Total  Time-  Subsurface  Targets  (TT-ST)  Looks  at  total  time  from  the 

completion  of  the  attack  strike  (on  subsurface  targets)  to  the  point  when  all 

BDA  assessment  and  dissemination  is  complete. 

time 

n/a 

4.  Package  Effectiveness  (PE) 

#  targets  killed 

#  of  packages 

5.  Package  Planning  Effectiveness  (PPE) 

#  targets  attacked 

#  of  packages 

6.  Attack  Effectiveness  (AE) 

#  targets  killed 

#  targets  attacked 

7.  Weapons  per  Target  Kill  (WPTK) 

total  #  of  weapons 
dropped 

#  targets  killed 
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Architecture 

Method  for  Metrics 
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Initial  system 
views  did  not 
capture  MOE’s 

Built  additional 
views  at  higher 
level  of  abstraction 
for  visibility  (ESD) 

Established 

Traceability 


MOEs  measured 
at  Perform  C2  Activity 


N  etc  wife  fn-rfonrswi  JFC 


MOPs  measured 
at  Perform  BDA 


RiaEatfc  £1?eeta. 
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Single  Package  Model 

Traceability  to  MOEs 
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■  Purpose:  Construct  analytical  model  based  on 
architecture  to  evaluate  the  WBBDA  system  concept 

■  Model  outputs  values  for  the  following  MOEs: 

■  Package  Planning  Effectiveness  (PPE) 

=  #  of  targets  attacked 

■  Package  Effectiveness  PE 
=  #  of  targets  destroyed 

■  Attack  Effectiveness  AE 

=  #  targets  destroyed  /  #  targets  attacked 

■  WPTK  =  #  weapons  used  per  target  destroyed 
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Single  Package  Model 

Key  Terms 
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■  Pk  -  probability  of  kill  (hit)  based  on  all  non-WBBDA 
factors  (weapon  performance,  delivery  system 
performance,  etc.) 

■  Accuracy  -  probability  WBBDA  correctly  determines  a 
hit  /  miss 

■  Reliability  -  probability  WBBDA  correctly  transmits 
and  displays  a  hit  /  miss 
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Single  Package  Model 

Scenarios 


m  AS-IS 

■  2  bombs  /  target,  simultaneous 

■  A/C  RTB  w /  0  bombs 

■  TO-BE:  WBBDA 

■  1  bomb  /  target,  repeat  until  WBBDA  “hit” 

■  A/C  RTB  w/  remaining  bombs 

■  Same  #  of  targets,  less  bombs 

■  TO-BE:  WBBDA  +  Doctrine  (W+D) 

■  DOT_LPF  doctrine  change  (WBBDA  +  drop  remaining  bombs 
on  additional/secondary  tgts) 

■  A/C  RTB  w /  no  bombs 

■  More  targets,  same  #  of  bombs 


17 


Single  Package  Model 

Example 
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■  Drop  1 00  bombs  on  1 00  targets 

■  Assume:  Pk  =  0.80,  Reliability  =  0.95,  Accuracy  =  0.90 


Drop 

100 

Bombs 


Weapon  Results 


20  miss  tgt 


WBBDA  Reliability 


WBBDA  results  on  76  hits 


No  WBBDA  results  on  4  hits 


WBBDA  results  on  19  misses 


No  WBBDA 

results  on  1  miss 
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Single  Package  Model 

Example  (cont’d) 


WBBDA  Reliability 


WBBDA  results  on  76  hits 


i-  Noi  WBBDA  results  on  4  hits 


WBBDA  results  on  19  misses 


No  WBBDA  results  on  1  miss 


WBBDA  Accuracy 


Correctly  assess  68  hits  as  hits 


Correctly  assess  17  misses  as  misses 


iKPrectlyia^^^BMises  as  hits: 


Dropped  100  bombs 
Lack  of  WBBDA  Results  95  bombs _ * 


(due  to  Reliability) 

No  WBBDA  results  = 
assumed  miss  on  5  tgts 


5  bombs 


State  ( 
Hit  Tgt 

Nature 
Miss  Tgt 

< 

Q  Assess  Hit 

68 

2 

CD 

co 

^  Assess  Miss 

8 

17 

19 
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Single  Package  Model 

Example  -  Targeting  Implications 


Results  of  1st  attack-implications  to  further  targeting 
(Pk=.8,  Rel.=.95,  Acc.=.9) 


Drop  100  bombs 
WBBDA  results  on  95 
No  WBBDA  on  5 


WBBDA  Results 


Lack  of  WBBDA  Results 

No  WBBDA  results 
assumed  miss  on(g  tgts 

Reattack  targets 
(80%  already  destroyed) 


- > 

State  < 
Hit  Tgt 

sf  Nature 
Miss  Tgt^ 

< 

Q  Assess  Hit 

Ijy 

00 

DO 

^  Assess  Miss 

(jlX 

I 


Type  I  Errors 

Retire  targets 
(targets  survive) 


Retire  targets 
(tgts  destroyed) 


Reattack  targets 
(all  missed) 


Type  II  Errors 
Reattack  targets 
(all  hit)' 
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Single  Package  Model 

Example  -  Overall  Results 
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■  Results  after  all  reattacks  (<  4  passes...  100,  30,  5,  2) 

■  Strike  package  departs  with  100  WBBDA  “hits” 

■  Overall:  97  targets  destroyed,  3  missed  (Type  I  Errors) 


State  of 
Hit  Tgt 

:  Nature 

Miss  Tgt 

Assess  Hit 

97 

3  Type  1 

Assess  Miss 

0  Type  II 

0 
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Single  Package  Model 

Actual  Results  w/  Inputs  at  Baseline 


INPUTS:  Components  WBBDA  Effectiveness  (all  baseline) 


Weapon  Pk 


0.80  0.051 


normal  distribution 


OUTPU 


Package 


Package 


Attack  Eff 


Weapons 


OUTPUTS 

As -Is 

WBBDA 

WBBDA  w/  doctrine 
change 

M 

<y 

P 

a 

% 

improve 

in  |i 

P 

O’ 

% 

improve 

in  u 

PPE  (planned) 

100 

0.0 

100 

0.0 

0.0% 

145 

0.0  i 

^  45% 

PE  (destroyed) 

95 

2.1 

98 

1.{ 

C  2.4% 

)  139 

4.7 

45% 

AE  (PE /PPE) 

0.952 

0.021 

0.975 

0.0' 

0 

2.4% 

0.956 

0.032 

0.3% 

WPTK 

2.10 

0.04 

1.41 

0.0 

_ 

V33V 

1.42 

0.09 

C  -33% 

_ ^ 

WBBDA 

WBBDA  +  Doctrine 


WBBDA  capabilities  improve  on  the  AS-IS  scenario 
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%  improvement  in  PE  =  # 
tgts  destroyed 


Single  Package  Model 

Sensitivity  to  Weapon  Pk 
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PE  Vs.  Pk  (%  improvement  relative  to  As-ls) 

80% 

60% 

40% 

20% 

0% 

0.50  0.55  0.60  0.65  0.70  0.75  0.80  0.35  0.90  0.95  1.00 

Weapon  Pk 


WBBDA  WBBDA+Ooctrine 
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Single  Package  Model 

Sensitivity  to  WBBDA  Reliability 


WPTK  Vs.  WBBDA  Reliability  (%  improvement  relative  to  As-ls) 


Supports  establishment/study  of  a  Reliability  requirement 
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Single  Package  Model 

Sensitivity  to  WBBDA  Accuracy 


PE  Vs.  WBBDA  Accuracy  (%  improvement  relative 

to  As-ls) 


WBBDA  Accuracy 


■WBBDA 


■WBBDA+  Doctrine 


0.79 


Supports  establishment/study  of  an  Accuracy  requirement 
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Single  Package  Model 

Aircraft  Loadout  Comparison 


■  Does  WBBDA  capability  favor  either  scenario? 

■  More  weapons  per  jet  of  lower  Pk  (SDB  scenario) 

■  Fewer  weapons  per  jet  of  higher  Pk  (JDAM  scenario) 


2,000#  JDAM 

250#  SDB 

500#  JDAM 

As-ls 

WBBDA 

W  +  D 

As-ls 

WBBDA 

W  +  D 

As-ls 

WBBDA 

W  +  D 

#Tgts  Destroyed 

78 

1.3% 

54% 

70 

8,6% 

33% 

78 

1.3% 

54% 

#  Bombs  Dropped 

160 

-34%  1 

0% 

160 

-19% 

-1% 

160 

[  -34% 

0% 

#Sorties  Flown 

80 

0.0% 

0% 

20 

0,0% 

0% 

40 

0,0% 

0% 

1  Optimum# of  Sorties 

1  80 

*34% 

0% 

20 

-16% 

0% 

40  1 

-33% 

0% 

|  Tgts  Dest./  Opt.  Sortie  | 

|  0.975 

[523% 

54%1 

3.6 

\  277% 

£  33% 

195 

IsoTT 

54% 

Analysis  of  model  results  forced  reconsideration  of 

MOEs,  architecture,  and  model 
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Architecture  Based  Evaluation  Process 

(ABEP) 


■  STEP  1 :  Design  Ops  Concept  (OV-1)  of  System  to  be  Evaluated 

■  STEP  2:  Identify  MOE’s  Relevant  to  the  Decision/Evaluation 

■  STEP  3:  Identify  Required  Level  of  Abstraction  for  Architecture 

to  Show  Traceability  to  MOE’s 

■  STEP  4:  Identify  Architecture  Views  Necessary  to  Capture 

Structure/Relationships.  NOT  VIEWS,  BUT  DATA 

■  STEP  5:  Develop  Architecture  Views  NOT  VIEWS,  BUT  DATA 

■  STEP  6:  Modeling/  Simulation  consistent  with  Architecture 

■  STEP  7:  Evaluate  Model  Completeness 

■  STEP  8:  Evaluate  MOE 
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Conclusion 

ABEP  vs  DODAF 


ABEP  Step  1 . 
OPS  Concept 


Step  2. 

ID  Mission 
Level  Metrics 


_ TA _ 

Determine  the 

intended  use  of  the 

architecture 

Step  3. 

ID  Required 
Level  of  Abstraction 
for  Traceability 


Step  4. 

Determine  Views  to 
Capture  Relationship 


Step  5. 

Develop  Architecture 
Views 


Step  6. 

Develop  Modeling 
Simulation 


Step  7. 
Evaluate 

Model  Completeness 


Step  8. 

Evaluate  Model  for 
MOE  Results 
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6  Step  DoDAF  vl.5 
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ABEP  Step  1 . 
OPS  Concept 


Step  2. 

ID  Mission 
Level  Metrics 


Step  3. 

ID  Required 
Level  of  Abstraction 
for  Traceability 


Step  4. 

Determine  Views  to 
Capture  Relationship 


Step  5. 

Develop  Architecture 
Views 


Step  6. 

Develop  Modeling 
Simulation 


Step  7. 
Evaluate 

Model  Completeness 


Step  8. 

Evaluate  Model  for 
MOE  Results 
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Conclusion 
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■  WBBDA  Specific 

■  WBBDA  +  Doctrine  Shift  significantly  increases  MOE’s 

■  WBBDA  Performance  is  sensitive  to  Accuracy,  Reliability,  &  Pk 

■  Non-WBBDA  Conclusions 

■  Architecture  can  be  used  to  effectively  evaluate  a  system  concept 

■  Evaluate  Gaps  (FNA)  and  Evaluate  Alternatives  (FSA  and  AoA) 

■  Identify  Critical  Requirements,  KPP’s 

■  Provide  Feedback  for  Architectural  Changes  &  Emerging  MOE’s 

■  Process 

■  Evaluation  w/o  Architecture  =  Inaccurate  Evaluation,  redundant 
effort,  non-Concordance 

■  Architecture  w/o  Evaluation  =  Static  Architecture 


Architecture  can  be  used  effectively  to  perform 
concept  definition  and  analysis  in  support  of  JCIDS 


4  / 
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Are  we  test  planning  differently  in  this  DoD 
network-centric,  system  of  systems 

environment? 


Are  we?  Should  we?  Can  we?  How? 
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Future 

Attributes 


T&E  Resources 


Analysis 


Observations 


Implications 


Recommendations 


Background 


•  Operational  concern: 

•  Air  Combat  Command  is  “enterprise  manager”  for 
AF  Distributed  Common  Ground  Station  (DCGS) 

•  Test  events  being  planned  without  coordination 

•  T&E  plans  not  validated 

•  Missing  opportunities  to  “piggy-back”  test  objectives 

•  Problem:  AF  not  yet  transitioned  from  system-centric  to  SOS 
approach  to  T&E 

•  Focus:  ACC  Force  Development  Evaluation  (FDE)  Process 

•  Methodology: 

•  Policy  and  Guidance  Review  (Policy) 

•  As-ls  FDE  Process  (Process) 

•  SYERS-2A  Case  Study  (Practice) 


Net-centric  trailblazer 


Services  jointly  pur  site  *Google’-like  mteUigeaffi-sharing  system 
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“System  of  Systems”  T&E 


Cliche?  No,  a  real  problem 

...  a  real  research  area 

DAU  Acquisiton  Guidebook: 

Defines  System  of  Systems  (SoS)  as  a  set  or 
arrangement  of  interdependent  systems  that  are 
related  or  connected  to  provide  a  given 
capability. 

SoS  Characteristics  (Maier  1996,1998) 

1.  Operational  Independence 

2.  Managerial  Independence 
Other  Characteristics 

Evolutionary  Development 
Emergent  Behavior 
Geographic  Distribution 


System  of  Systems 
Systems  Engineering  Guide: 
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9  SoS  Integration  Lessons 


MS.  HR  FORGE 


1.  Activities  n e  Ed  to  p r seed e  SoS  i  nte  g ration 

■  Architecture,  system  irtteiface  tests.  Architecture  compliance 
Z.  Early,  increm  e  ntal  and  ite  rative  i  ntegration 

&  Robust  testing  strategy 

4_  Plan  for  substantial  difficulties  and  significant  time  and  resources 
&.  Use  of  a  single  facility  facilitates  integration  of  SoS  components 
&,  Address  the  leadership  of  the  integration 
T.  heed  for  common  processes  and  infrastructure 

■  Engineering  boards,  tracking  requirements,  SoS  issue  ID, ... 
a.  Effective  common  processes 

■  daily  planning,  timely  dissemination  of  info,  status  meetings 

i.  Prototyping  the  SoS  provides  early  insight  in  the  ops  requirements 
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V  Test  Policy/Guidance  Review  * 


•  Public  Law,  DoD  Policy 

•  AF  Guidance 

•  AF  Policy  Directive  (AFPD)  99-1 :  T&E  Process 

•  AF  Instruction  (AFI)  99-103:  Capabilities  Based  T&E 

•  “Seamless  Verification” 

•  Integrated  Test  Team  (ITT) 

•  Common  T&E  Data  Management  (Open  Database) 

•  Air  Combat  Command  Instruction  (ACCI)  99-101 

•  Other 

•  Defense  Acquisition  Guide  (DAG) 

•  International  Council  on  Systems  Engineering  (INCOSE) 

•  ANSI/EIA-632 


V 
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Test  Policy/Guidance  Review 


Air  Combat  Command  Instruction  (ACCI)  99-101 : 
Test  and  Evaluation 

•  Electronic  Project  Order  (EPO) 

♦  Test  Priority  List  (TPL) 


Credible  OT 


s 

n>  - 

I-  ^ 


Others 

♦  AFT&E  Guidebook 

•  53rd  WG  Test  Team  Handbook 


Info  Req’d 


E 

U\ 

Resources  Fenced 

in 

Q 

□ 

For  Full  OT.  if  required 

—  _ ikA 

£ 


Operational 
Test  Design 


Adequate  DT 


Info  Reel’d 

Ini 

o  R 

:eqrd 

1 

1 

I- 

ui  &l  met  oy 
DT/OT 

I  n  teg  rated  Test 
Design 


Developmental 
Test  Design 


DT  +  OT  =  Integrated  Lifecycle  Test  Focus 
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SoS  Test  Guidance 


V 


•  Defense  Acquisition  Guide  (DAG)  -  Chapter  9 

An  important  aspect  is  to  develop  a  strategy  for  testing  each 
system  in  the  context  of  the  system-of-systems,  or  family-of- 
systems  architecture  within  which  it  is  required  to  operate. 

The  shift  away  from  point-to-point  system  interfaces  to 
network-centric  interfaces  brings  implications  for  the  T&E 
community. 
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SoS  Test  Guidance  Review 


•  INCOSE,  Systems  Engineering  Handbook  (ver  2a) 

•  System  Integration  with  External  Interfaces 

•  ICDs,  Interface  working  Groups 

•  Review  test  procedures  and  plans  which  verify  these  interfaces 


Analysis  Requests,  Requirements,  Implemented  Products 

o 


ANSI/EIA-632,  Processes  for  Engineering  a  System 

•  Technical  Evaluation:  Analysis,  Verfication  and  Validation 

•  Application  Context 

-  Enterprise  Factors 

-  Enterprise  Support 

-  External  Factors 

-  Other  Enterprise  Projects 


TT 

Analytical  Models  &  Assessments,  Validated  Requirements, 
Verified  System  Products,  Validated  End  Products 
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Air  Force  Test  Policy 


& 


•  Observation  1: 

A  Shift  to  Integrated,  Capabilities-Based  T&E 

•  Observation  2: 

Seamless  Verification  Still  Has  Seams 


Milestone  B  Decision 


Milestone  C  Decision 


Develop  and 
Demonstrate 
System  -  DT 

1 


^  A1 


Demonstrated  System  / 


Contractor  and  SPO 


Context  for  Force  Development 
Evaluation  (FDE)  Process: 

DoD  Acquisition  System 


Produce  and 
Deploy 
System  -  OT 


Deployed  System 


A2 


AFOTEC 


'ained  System 


ACC  T est  Ce 


anizations 


W  ForCe  Devel°Pment  Evaluation 
J* (FDE) 

•  A  Subset  of  Operational  Test  and  Evaluation  (OT&E) 

•  Demonstrate  the  operational  effectiveness  and 
suitability  of  a  system  as  evolutionary  upgrades  are 
made  to  sustain  its  relevance 
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Discrepancy  Reports 


FDE  Process  Observations 


•  Observation  3: 

Apparent  Lifecycle  Seams  are  Mitigated  by 
Cooperation  in  the  T&E  Community 

•  Observation  4: 

Seams  Among  Interdependent  Systems 
are  Real 

•  Observation  5: 

Integration  is  NOT  Built  Into  the  Process 


System  Modif ication 


Modification  Info 


Electronic  Project  Order  Revisions 


Generate  FDE 


Developmental  T&E 


Afe.2.1 


Electronic  Project  Order  for  Action 


A  5.2.2 


MAJCOM  Test  Center  Organizations 
MAJCOM  Staff 

Test  Priority  List  Integrated  Product  Team 
Combatant  Commands 
Other  Testing  Agencies 
MAJCOM  Lead  Staff  Agency  for  T&E 
AF  C2,  Intelligence,  Surveillance  &  Reconnaissance  Center 


Conduct  FDE  Process 


Validated  FDE  Plan 


FDE-Ready  System 


MAJCOM  Operational  Resources 


FDE  Case  File 


Case  File  w/Results 


End-of-FDE  Message 


Fielding  Recommendation 


Discrepancy  Reports^ 


FDE  Report^ 


Recommendations  in  MAJCOI^I  Database 


Closed  Case  File  ^ 


Operational  Briefings  , 


MAJCOM  Operational  Units 


System  Modification 


FDE  Process  Observations 


•  Observation  6: 

FDE  Process  Accommodates  SOS 
Testing  But  Doesn’t  Deliberately  Force  it 

•  Observation  7; 

Resource  Constraints  Limit  ACC’s  Ability  to 
Develop  SOS  FDEs 

•  Observation  8: 

Process  is  Beginning  to  Embrace  Non- 
Traditional  Weapon  Systems 


Order  for  Action 


FDE  Process  Observations 


•  Observation  9: 

Increasing  Load  on  the  FDE  Process 

DoD  T&E  Summit  2004,  Dr.  Glenn  Lamartin: 

•  From  platforms  to  capabilities  &  SOS  solutions 

•  Increasing  complexity  and  interdependencies  of  systems 

•  Exponential  growth  in  interfaces  (network  participants) 

•  Increased  requirements  for  T&E  (Evolutionary  Acq) 

NCW,  Alberts,  Garstka  and  Stein 

“Testing  systems  will  become  far  more  complex  since  the  focus 
will  not  be  on  the  performance  of  individual  systems  by  on  the 
performance  of  the  federation  of  systems” 
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System  Modification 


AF  C2,  Intelligence,  Surveillance  &  Reconnaissance  Center 


MAJCOM  Operational  Units 


Electronic  Project  Order  for  Action 


FDE  Process  Observations 


•  Observation  10: 

Test  Center  Project  Manager  (PM)  is  the  Key 
Actor  in  FDE  Planning 

•  Observation  11: 

Lack  of  AF -Level  Guidance  on  T&E 
Information  Management 
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Sensor  Case  Study 


•  Platform:  U-2S  -  high  altitude  surveillance  &  reconnaisscance 

•  Sensor:  SYERS-2A  -  multispectral  (EO/IR)  imaging  sensor 

•  Upgrade  to  airborne  processor  with  ATM  interface 

•  Data  Link:  Dual  Data  Link  2  (DDL  2-LOS  and  BLOS  configurations 

•  Ground  Station:  AF  DCGS  -  dispersed  ground  systems  supporting  first-phase 
analysis  of  U-2,  Predator,  Global  Hawk  and  other  sensors  via  secure  WAN 
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Numerous  Stakeholders 


■ftJQL  & 


...  an  insightful  OV-4 


Enterprise  Management  Test  Planning 

Requirements  Test  Execution 

Test  Resourcing  Airborne  and  C2 

Test  Coordination 


Operations 
Air  and  Ground 


25 


Numerous  Stakeholders 


...  an  insightful  OV-4 


DCGS  Sustainment  (O&M) 

U-2  Sustainment  (O&M) 

DCGS  System  Program  Management 
New  Acquisition  and  Modernization 

U-2  System  Program  Management 
New  Acquisition  and  Modernization 

Flight  Test  Facility 
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Sequence  of  Events 


Test  Objective:  “Verify  SYERS-2A  sensor  end-to-end  operations  and  to 
demonstrate  full  airborne/ground  segment  functionality  with  DLL2  in 
available  configurations  and  operational  representative  architectures” 


A2YD  contacts  Det  2  in  an  effort 
to  coordinate  future  FDE  activity 


A2YD  sees  FDE  plan  and  expresses 
concern  about  absence  of  coordination 


A3YR  unilaterally  announces  start 
of  SYERS-2A  FDE  in  late  Oct 


U-2  FTF  e-mail  prompts 
A2YD  concerns  about  lack 
of  FDE  awareness 


Det  2  staffs  SYERS-2A  FDE 
plan  through  the  53  WG  chain 


AF  DCGS  ITT  meets;  focus  is  on  AF 
DCGS  modernization  vice  upcoming 
“end-to-end”  FDEs 


Det  2  queries  A2XD  for  POC 
at  605  TES;  shares  timeline 
for  U-2  S  FDEs  with  A2YD 


Analyzed  message  traffic,  documents,  and 
vast  discussions  with  SME/  POCs 
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Complex  Interactions 


AF  DCGS  Sustainment  Guidance 


V  Case  Study  Observations 


•  Observation  12: 

Program  Priorities  Dominate  Even  Among 
Interdependent  Systems 

•  Observation  13: 

System-Centric  Management 

•  Observation  14: 

System  Focus  for  the  Fielding  Decision 

•  Observation  15: 

Some  Coordination  Tools  Left  Unused 

•  Observation  1 6: 

Ability  to  Define  the  “Ends”  Disappearing  as 
Net-Centric  Reality  Emerges 
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Implications  for  T&E 


& 


Implication  1:  Co-evolution  Is  Critical 


Exposure  to  new  information  technologies  and  their 
capabilities  is  potentially  dangerous  unless  it  is 
accompanied  by  changes  in  a  number  of  key 
dimensions. 

-  Alberts,  Information  Age  Transformation 


Doctrine 

Training  &  Education 


Test  &  Evaluation 


Implications  for  T&E 


Implication  2:  End-to-End  Is  Out,  Net-Ready  Is  In 


m  >> 


Focus  of  test  and  evaluation  needs  to 
shift  from  the  performance  of  individual 
entities  to  their  ability  to  add  value  to  the 
networked  force. 


-  Alberts,  Information  Age  Transformation 


T&E  Resources 


Implications  for  T&E 


Implication  3:  SOS  T&E  Can’t  Work  Alone 


SOS  T&E  should  complement  a  strategic  planning,  budgeting, 
requirements  development,  and  acquisition  system  fundamentally 
oriented  toward  generating  enterprise/mission  capabilities  instead 
of  individual  systems. 
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Recommended  Characteristics 
**  for  future  SoS  FDE 


1 .  Scope  to  Validate  Operational  Capabilities 

•  How?  Use  DoDAF  Products/  M&S  to  understand 
complex  relationship  of  systems  and  capabilities 

2.  Use  Net-Readiness  Objectives  to  Validate  SoS 
Interoperability 

♦  How?  Use  DoD  Net-Centric  Data  Strategy: 


Visible 

Agile 

Accessible 


T  rusted 

Responsive 

Understandable 


3.  Prioritize  According  to  Operational  Risk 

4.  Employ  appropriate  Integration  Environments 


Conclusion 


•  Policy  and  guidelines  now  reflect  the  changing  IT 
landscape  of  system  of  systems. 

•  Integrated  T&E  and  Seamless  Verification 

•  Leaders  have  predicted  this  changing  landscape  will 
directly  impact  T&E  activities 

•  Lessons  can  be  learned  from  enterprise  case  studies 

•  Many  organizations/  enterprises  may  rely  on  the 
heroics  of  system-level  test  managers  to  handle  this 
added  SOS  focus 


Changes  to  Integration,  Test  and  Evaluation  in  a 
network-centric  SoS  environment  is  imperative 
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Case  studies:  A  common 


3  Robotics 

Research  Labs 
Robotic  Embedded  Systems  Lab 


Language  Between  Engineers  and 

Managers 

10th  Annual  NDIA  Systems  Engineering 

Conference 

DeWitt  T.  Latimer  IV,  dlatimer@usc.edu 

http  ://www .  ro  boti  cs .  u  sc .  ed  u/~d  I  ati  m  e  r 
Center  for  Systems  and  Software  Engineering 

http://csse.usc.edu 

Robotics  Embedded  System  laboratory 
http://robotics.usc.edu 

Viterbi  School  of  Engineering 
University  of  Southern  California 


The  views  expressed  in  this  presentation  are  those  of  the  author  and 
do  not  reflect  the  official  policy  or  position  of  the  United  States  Air 
Force,  Department  of  Defense,  or  the  U.S.  Government. 
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•  Background 

-  Communication  Facilitation 

-  Engineering  training  and  education 

-  Manager  training  and  education 

.  Possible  Discourse  Problems 
.  Example  Case  Studies 

-  Surface  Assessment  Robot 

-  Autonomous  Helicopter 

•  Observations  &  Pitfalls  Experienced 
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Background 
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•  Review  communications  basics  -  where  would 
discourse  about  cases  help? 

•  Review  education  and  training  of  engineers  and 
managers  to  establish  a  baseline  of  what  each 
community  is  comfortable  communicating 
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Communication  starts  with  understanding. 

-  R.  Kline 
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Has  this  happened  to  you? 


“What's  wrong  with  this  picture?” 
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Communication  Stages 


Receive  !  Largely  a  physical  (sound)  or  technical  (email)  phenomenon 


Attended  !  Did  the  recipient  pay  attention  to  the  message  (raise  to  their 
consciousness,  open  the  email)? 


Understood  r\Did  the  recipient  form  the  desired  mental  concepts? 


Responded  /f  Did  the  recipient  confirm  understanding  or  was  the 

"act  on  the  understanding? 


Remembered  :  Did  the  recipient  commit  the  facts  to  memory? 
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Engineering  Training  and  Education 


•  Emphasis  on  models,  accuracy,  precision,  and 
addressing  uncertainty  as  a  statistical  quantity 

-  Gather  data  from  many  projects/cases  to  integrate 
into  models 

-  Apply  the  data  collected  to  enhance  models 

-  Learn  rules  of  how  to  accurately  apply  models  to 
projects 

•  Emphasis  of  engineer  training  is  the  concept  of 
“due  care”  in  the  generation  of  products  - 
accepting  that  sometimes  things  “just  happen” 
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Management  Training  and  Education 


•  Most  programs  heavily  involve  case  studies  that 
illustrate  quantitative  models  in  action 

-  The  cases  provide  “grounding”  as  to  where  the 
models  are  valid  and  how  to  utilize  them 

-  Indeed,  a  tenant  from  many  management  students 
is  that  models  may  be  easily  invalidated  by  moving 
to  a  different  set  of  environmental  factors 

.  And  there  are  case  studies  to  illustrate  this 

.  The  focus  on  management  training  is  to  identify, 
prevent  if  possible,  and  report  on  things  that 
may  disrupt  the  manager's  span  of  interest 
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•  My  experience 

•  Anatomy  of  a  disagreement 
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From  my  experience... 

•  Managers  want  to  know  if  the  model  truly 
represents  the  problems  they  are  about  to 
encounter,  or  that  the  model  gives  them 
information  about  how  to  handle  the  problems 
without  wasting  resources 

.  Engineers  prefer  problem-relevant  models  in 
which  they  have  experience  and  desire  to  use 
them  in  the  fashion  in  which  they've  been 
trained 
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•  Conflict  arises  when  the  engineer  and  manager 
are  in  disagreement  -  e.g.  “Can  we  produce  a 
system  at  the  20%  confidence  effort  estimate?” 

.  Different  views  of  the  data  is  a  possible  cause 

-  Manager  believes  that  the  uncertainty  is  the 
management  trade  space 

-  Engineer  believes  the  uncertainty  is  the  inherent 
variation  in  performing  the  tasks 

.  Both  may  be  correct!!  How  do  we  begin  a 
rational  discourse? 
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Example  Case  Studies 


In  the  following  two  cases,  we  will  cover  two 
examples  of  where  schedule  for  production  of  a 
robotic  system  failed  to  achieve  a  usable 
system  on  time 


.  Surface  Assessment  Robot  -  met  all  stated 
requirements,  unstated  requirements  cause  system  to 
fail  to  integrate  with  all  stakeholders 

•  Global  Hawk  -  short  engineering  /  manufacturing 
design  phase  led  production  problems 
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Surface  Assessment  Ro 


Project  involved  numerous  precedented 
technologies  in  construction  assessment 


-  Unprecedented  nature  of  fully  taking  an 
engineer  out  of  the  loop  created 
requirement  development  risk 
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Global  Hawk 


.  System  carries  remote  sensing  payloads  under 
guided  flight  autonomy  command  of  remote  operator 
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.  Reaction  when  using  cases  as  the  basis  to 
discussion  between  engineers  and  managers 

.  Example  outcomes  of  managers  engaging  and 
investing  in  engineering  decisions  flowing  out 
from  case  study  dialog 

.  Strengths  and  weaknesses 
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.  Managers  who  were  later  exposed  to  these 
cases  engaged  to  address  the  root  causes 

-  Many  proposed  additional  forward  looking  models 

-  Many  engaged  on  setting  estimation  parameters  in 
constructive  effort  models 

.  Managers  began  to  see  where  the  “trade- 
space”  was  outside  of  the  confidence  intervals 

-  Expressed  understanding  of  the  inherent  risk  in 
various  parameterized  descriptions  of  the 
environment 
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.  Using  the  surface  assessment  robot  case, 
managers  brought  up  how  strategic  alignment 
should  be  linked  to  various  effort  model  settings 

.  Using  the  Global  Hawk  UAS  case,  managers 
mentioned  the  importance  of  having  reliable 
assessment  metrics  for  maturity  of  engineering 
products  in  addition  to  the  processes 

-  Identified  the  lack  of  ability  to  assess  reuse 

-  Identified  the  lack  of  upfront  investment  to  ensure 
work  products  could  be  reused  and  need  to  invest 
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Case  Studies  typically  do  not  have  “schoolhouse” 
answers  -  this  is  their  strength  and  weakness 


-  Strength  -  the  case  describes  the  breadth  of  what 
happened  and  the  environmental/political  factors 

-  Weakness  -  the  case  doesn't  tell  you  what  to  do  if 
you  aren't  doing  the  exact  same  thing  in  the  exact 
same  situation  and  time 
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Cases  as  a  Common  Language 


Cases 


24/10/2007 

Background 


Managers 


Discourse 


■ 

■ 

on  t 

■  :J 

c 

♦ 

=2 

s 

♦With  SP 

Qlntermedi  ate 

HSF  concretes 

□  Without  SP 
■  Slump  <  80  mm 

4 

♦ 

logit  [Pr^l  =  Ad 

+  Yirz«> 

+  (Xcj—Xc)(d*f) 

+  r 

w 

where 

( X.-Y)  - 

if-4*-1 

0  otherwise 

and 

if  dog- q 

\X0  x  <}(.«*) 

0  otherwise 

'V/V' 

Engineers 

Observations 


Slide  19 

Pitfalls 


I  III  I 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


Pitfalls 


“Robotics 

|  Research  Labs 
Robotic  Embedded  Systems  Lab 


Two  pitfalls  I  have  encountered  when  using  cases 
to  support  engineering  positions  with 
management 

-  I  want  more  cases  to  support  your  position... 

-  We  have  to  use  the  model  based  result,  even  if  it 
doesn't  make  sense! 


...  and  what  can  be  done  about  these  pitfalls 
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“That's  only  one  (two,  three...)  case  that  supports 
your  model,  now  give  me  more  cases!” 

-  Likely  that  you  may  only  be  conversant  in  a  few 
cases  relating  to  any  one  set  of  circumstances 

-  This  may  be  done  if  someone  is  engaging  in 
selection  bias  for  which  cases  they  consider  for 
when  forming  their  theory  of  the  solution 

-  Remember,  calibrated  models  are  the  integration  of 
many  observations,  where  each  observation  is  a 
case  explained  in  the  parameters  of  the  model! 
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“The  model  says  something  we  didn't  expect,  we 
have  to  use  the  results  because  its  the  model” 

-  “Expect”  is  based  on  understanding  of  this  specific 
project  in  progress  and  similar  historic  cases 

-  Model  may  be  leveraged  beyond  its  calibration  or 
model  parameters  do  not  enable  discrimination 
between  cases  with  different  outcomes 

-  People  should  examine  some  relevant  case  to  see 
how  well  the  model  is  calibrated  to  this  problem  or 
look  for  other  factors  that  could  impact  the  result 
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•  Background 

-  Communication  Facilitation 

-  Engineering  training  and  education 

-  Manager  training  and  education 

.  Experiences  Using  Case  Studies  in  Discourse 
.  Example  Case  Studies 

-  Surface  Assessment  Robot 

-  Autonomous  Helicopter 

•  Observations  &  Pitfalls  Experienced 
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The  goal  is  effective  communication 
through  engaged  listening  and  speaking. 
Motivating  cases  may  be  one  avenue  to 
appropriate  understanding  and  responses 

in  multidisciplinary  teams. 
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Additional  Sources  of  Case  Studies 


Defense  Acquisition  History  Project 

http://www.army.mil/cmh/acquisition/research/fa_casestudybib.html 

AFIT  Case  Studies  web  page 

http://www.afit.edu/cse/cases.cfm 

The  Risks  Digest 

http://catless.ncl.ac.uk/Risks/ 

Systems  Engineering  Handbook 

Available  from  INCOSE.org 

Technical  Project  Management  Textbooks 

one  example:  Kermer,  “Software  Project  Management:  Readings 
and  Cases”,  McGraw  Hill,  1 996 


24/10/2007 


Slide  26 


I  III  I 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


References 


“Robotics 

|  Research  Labs 
Robotic  Embedded  Systems  Lab 


Brooks,  “The  Mythical  Man-Month:  Essays  on  Software  Engineering”,  1st  edition, 
Addison-Wesley,  200  pages,  1978. 

Kline,  “Listening  Effectively”,  Maxwell  Air  Force  Base,  Ala.:  Air  University  Press,  1996. 

Latimer,  “Acquiring  and  Engineering  Robotic  Systems”,  Qualification  Exam  Report,  USC 
Center  for  Systems  and  Software  Engineering  Technical  Report  number  USC-CSSE- 
2007-709,  May  2007. 

Yin,  “Case  Study  Research:  Design  and  Methods”,  3rd  Edition,  Sage,  2003. 

“Software  Engineering  2004,  Curriculum  Guidelines  for  Undergraduate  Programs  in 
Software  Engineering”,  the  Computer  Society,  IEEE,  ACM,  August  2004. 

“Eligibility  Procedures  and  Accreditation  Standards  for  Business  Accreditation”,  AACSB 
International,  January  2007. 


24/10/2007 


Slide  27 


I  III  I 


University  of  Southern  California 

Center  for  Systems  and  Software  Engineering 


About  the  Author 


“Robotics 

|  Research  Labs 
Robotic  Embedded  Systems  Lab 


Capt  DeWitt  Latimer  IV,  USAF  is  currently  assigned  as  a 
PhD  Student  to  the  Air  Force  Institute  of  Technology 
and  working  towards  a  PhD  in  Computer  Science  at 
the  University  of  Southern  California.  He  is  advised  by 
Prof  Barry  Boehm  and  Prof  Gaurav  Sukhatme.  His 
research  focuses  on  investigating  the  nature  of 
acquiring  autonomous  robotic  systems.  He  earned  his 
MS  degrees  in  Robotics  (2001)  and  Civil  Engineering 
(2002)  at  Carnegie  Mellon  University.  He  is  a  senior 
member  of  the  IEEE  and  ACM  and  a  member  of 
ASCE,  and  AFCEA  and  was  awarded  the  CSDP 
credentials  from  the  Computer  Society. 


24/10/2007 


Slide  28 


<\sset-Based  PBL  for  Navy  Warships  ~  A 
case  study  for  LCS  Class  Shi  pi 


O 


NDIA  -  10cn  Annual  Systems  Engineering 

Conference 

Oct  24,  2007 


Mike  Mahon 


j  Definitions/PBL  seeling 
•  LCS  Overview 
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Asset-Based  PPL 
Asset-Based  PPL 
Asset-Based  PEL 
Path  ahead 


Key  questions 
challenges/ Obsta  cles 
-  keys  to  success 
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Definitions 


What  Is  PEL? 

j  any  contract  where  the  primary  requirement  is  to 
provide  products  &  services  bused  on  a  pre¬ 
determined  performance  metric. 

j  The  performance  metric  should  in  some  way  be  a 
con  tributing  factor  to  Operational  Availability  (Ao). 


The  Navy  today  boosts  of  150 +  PEL  con  trusts;  most 
of  these  ore  supply-oriented  PELS  Issued  by 
NA  VICE 

•  Most  lire  lower  le  vel  component  bused  PELs 


Definitions  (coni) 


What  is  Asset-b  sed? 


Component  Level 


Easier 


Complexity  is  based  of  number  of 
systems  within  asset 


- ► 

Harder 


PBL  Complexity 


There  sure  two  primary  dimensions  to  PBL  complexity. 


PBL  Complexity  Scaling 


Note:  this  scale  is  Mahon  developed  and  not  an  industry  accepted/certified  rating  system  for  PBLs 


-Go  system  or  Systems 


LCS  consists  of  core  seaframes  design  ad  to  /josi 
mission  packages.  Three  MP  are  initially  planned. 


MIW 


VTUAV  (2) 


MH-60S 


COBRA  (2) 


ALMDS 

OASIS 


SCULPIN 


USSV  (1) 


BPAUV  (2) 


ASW 


MH60R  (1) 

3gt. 


onobuoyM«54 
Torpedo 


ALFS 


Multi-Static  Lightweight  Array 
Active  Source 

■RMV  (2) 


RTAS 


MFTA 


ASUW 


VTUAV  (1) 

% 


0  Helo 


EO/IR  GAU  21  Hellfire 
Gun 


NLOS-LS  (4) 


1  IK  HI |u 
1 


30mm  Gun  (2) 


Requirements 

THRESHOLD 

OBJECTIVE 

Sprint  Speed  (kts) 

40 

50 

Mission  Package  Payload 
(mt) 

180 

210 

Range  @  Transit  Speed 
(nm) 

3500 

4300 

Navigational  Draft  (ft) 

20 

10 

Core  Crew  manning 

50 

15 

Two  Ysaars  S220M 

Any  Mission  Package  /  Any  Ship  /  Any  Time 


•  Non-traditional  hull  forms 

•  Non-traditional  materials 

•  Non-traditional  Propulsion 

-  CODAG  +  Waterjet  drive  (x4) 

•  Non-traditional  construction  practices 

•  Non-traditional  system  suppliers 

•  Modular  Open  Systems  Approach 

•  Open  Computing  Architecture 

•  Automation 


LCS  Breaks  thru  many  Traditional  Paradigms 


LCo 


;  Flight  0  Acquisition 


6  Industry  Concepts 

(06  Feb  03) 


February 

2003 


3  Preliminary  Designs 

(19  Jul  03) 


July 


2  Final  Designs 

(Contracts  Awarded  27  May  04) 


} 

_ 


FLT  0  Lockheed 
Martin 
(15  Dec  04) 


May  December 


2003 


2004 


2004 


FLT  0  General 
Dynamics 

(14  Oct  05) 


October 

2005 


First  Ships  Delivers  in  Summer  2008 


LCS  Flight  0  Sustain  merit  Strategy 


The  US  Navy  approach  for  LCS  sustainment  is  to 
establish  the  held  shipbuilding  hums  as  hud  for 
sustainment  for  mi  interim  36-month  period. 

—  Concept  is  to  leverage  know! edge  for  design/ construct! on  for  risk 
mitigation  in  initial  sustainment  phase 


1.  Will  it  work?  Just  because  it  works  at  lower  level-: 
doesn't  mean  it’s  a  good  thing  at  a  higher  level? 


2,  Will  it  save  m  on  ey  an  d  if  soj  how  mu  oh  ? 


J. 


Cun  we  really  put  such  heavy  responsibility  for 
our  nations  defense  in  the  hands  of  Industry? 


4,  What  is  the  fallback  If  it  doesn't  work? 


5.  What  will  happen  to  the  existing  Infrastructure  that 
Is  still  repaired  for  other  ship  classes? 


Asset-based  PBL  -  Oharenges/Obstaclei 


1 

2 


LJ  j 


A 

SJ 


o. 


A-> 

o. 


Jobs/responsibilities  —  sue  ettuchmd 

Risk —  nil  dimensions  of  risk  mus  t  be  identified  end 

mitigation  plans  established  and  funded 

Colors-of-money  (RDT&E,  SCN,  O&M,  mud) 

•  /f  /s  difficult  securing  on  extra  SCN  dollar  to  save  two  d oilers  of 
OM&N 

Cost/ business  esse  ana  1  yses  (B  lj\ J 

•  Most  transformational  concepts  require  e  BCA,  yet  establishing 
e  baseline  for  today’s  worships  is  difficult  at  best 

Interaction  with  other  existing  BBLs 

•  deed  to  ensure  that  upper  level ,  asset-based  PBLs  con  work  In 
harmony  with  existing ,  established  PBLs 

Patience  (or  hick  of  it) 

•  Initial  performance  will  be  bumpy/full  of  glitches  -  ell  peril es 
need  to  be  prepered  for  this  end  work  through  it 


Asset-based  PBL  -  Keys  to  Success 


1 

2 


'j 


// 

-y« 


Support  from  DoD/Customer  community 
A  Good  approach  that  manages  Risk 

•  ‘stair-step9  approach  that  progresses  to  full  asset-based 
PBL  incrementally 

•  Initially  costs  mors  to  have  parallel  paths  In  csss  of 
failure 

•  Integrated  industry- Go  vt  proa 

So/icf  team  Structure 


•  Embraces/uses  competition  fur  optimal  value 

Good  performance  metrics 


Path  Ahead 


1.  Build  the  team  and  die  processes  for  the  three 
your  Interim  Sustainment  timeframe. 

2 Establish  initial  metric  set 

3,  Do  NOT  accept  PBL  from  Initial  suppliers  - 
risk/cost  will  be  too  high.  Instead  use  the  3yr 
period  to  understand  the  ship  and  operational 
caps  and  lims  “  measure  everything 1 

4.  Build  alternate  suppliers  —  keep  competitive 


v. 


en  vironment 

Establish  transition  plan  for  full  life-time  support 
(Also  build  phm  to  fallback  to  traditional  approach  If  reqd) 


Back-up 


Job/Responsibilitm 


The  biggest  o bstacie  to  asset-b  used  pel  [or  Esc 
or  CL  S)  will  be  from  the  organic  support 
infrastructure  who's  very  livelihood  is  threatened 
by  this  initiative 

Unlike  component  based  PEL  f which  never  shifted 
'who  did  the  work  but  how  it  was  contracted), 
asset-based  PBL  transitions  organic  responsibility 
to  industry 

And  yet,  industry  must  work  with  those  very  sum e 
organic  activities  to  develop  end  operate  the 
Asset-based  PEL 

Many  people/organizations  will  be  very  happy  to 
see  asset-based  PBL  hill  end  may  even  work  to 
help  it  hill 


Asset-Based  PBL  -  Org  structure 

•  The  business  structure  of  an  asset-based  PBL  for  a  worship  con  be 
very  complex .  It  consists  of  many  suppliers  end  varying  levels 


Govt  Program  Office 

♦ 


PSI 


l 

Sub  tier 
Integrator 
#2 

System 

3 

System 

4 

r  i 

I 


Sub  tier 
Integrator 
#1 

T 


System 

1 A 

System 

2A 

System 

IB 

System 

2B 

System 

N 

System 

N 

PSI  -  Supplier  level  PBL  integration 


N 


i 

k  i 

r  i 

k  i 

r  i 

k  i 

r  i 

k 

r 

Other 

Supplier 

Other 

Supplier 

Other 

Supplier 

Other 

Supplier 

40%-direct  supplier  PBL 


60%-PSI  provided  PBL 


Performance  Metrics 


j  Measuring  performance 
is  critical 

SCM 

Maintenance 

Training 

•Inventory  management 

•Casualty  response  time 

•Train-to-qualify  (T2Q) 

°  Samples  mom  os 

•Demand  forecasting 

•Remote  monitoring 

•Embedded  training 

include: 

•Transportation 

•Condition-based  Maint. 

•lnitial&  replenishment 

•Requisition  processing 

•Distance  support 

crew  training 

•Parts  Repair 

•‘O’  level  maint.  PM/CM 

•Computer  based 
training  &  sim 

•Parts  replenishment 

•‘1’  level  Maint.  PM/CM 

•Trainer  site  ops 

•SCM  management 

•‘D’  level  Maint 

•Team  training 

•Maintenance  Mgt 

•Training  management 

°  To  achieve  asset-based 
PBL  -  m  time  metric 
quantity  lessens  but  the 
metric  ‘ quality 9  grows 


Ground  Support  Systems 

Integrated  Defense  Systems 


Integrated  Diagnostics  (ID) 
Closed  Loop  Knowledge 
System  (CLKS) 

Steve  Head 


BOEING  is  a  trademark  of  Boeing  Management  Company. 
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Objective 


ID  CLKS 


■  Develop  ability  for  ID  engineer/analyst  to  gain  domain  knowledge 
from  integrated  data  stores 

■  Develop  closed  loop  knowledge  system  where  data  is  presented 
and  exploited  to  actively  influence 

-  Authoring/monitoring/adjusting  of  smart  diagnostics 

-  Engineering/analyst/maintenance  technician  judgment 

■  Maximize  use  of  current  transactional  databases,  domain 
experience  and  past  successes  on  aircraft/system  programs 

■  Significantly  improve  sharing  and  integration  of  related  information 
across  business  disciplines  to  enhance  decision  making  processes 

■  Utilize  results  and  lessons  learned  from  previous  Boeing  ID  data 
mining  studies  (2001  and  2002)  to  better  the  outcome  of  ID  CLKS 
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Aircraft 


Organizational 


SRA  /  SRU 


Cone  of  Tolerance 


Detect  fault  before  Pilot 


Maintenance  BIT 
Supplemental  Tests 


WRA / LRU / LRM 


Testing  at  these 
levels  duplicate 
failures  from 
upper  levels) 
of 

test. , 


Identify  all 
Failure  Modes 
(Utilize  FMEA) 


BIT  information  and 
aircraft  environmental 
conditions  are  captured 
by  the  aircraft. 


pilot  distinguish 
between  a  fault,  false 
alarm,  and  normal 
operation? 


Does  BIT 
detect  &  isolate 
all  failure  modes? 
Are  supplemental 
tests  required? 


Aircraft  debrief 
properly  interprets  BIT 
and/or  Pilot  observable 
faults. 


Does  the  design 
(Weapon  System  & 
Support  System)  isolate  to 
1  WRA  /  LRU  /  LRM? 
(BIT,  Tech  Pubs,  SE,  etc.) 


Supplier  testing  and/or 
Organic  testing 
addresses  all  failure 
modes? 


Provide  WRA  /  LRU  / 
LRM  tester  with  fault 
information  from  the 
Organizational  Level. 


Does  the  design 
(W eapon  System  & 
Support  System)  isolate  to 
1  SRA  /  SRU  ? 


Supplier  testing  and/or 
Organic  testing 
addresses  all  failure 
modes? 


Provide  SRA  /  SRU 
tester  with  fault 
information  from  the 
WRA / LRU / LRM 
tester. 


Does  the  design 
(Weapon  System  & 
Support  System)  isolate  to 
a  small  number  of 
components  ? 


Maturation  ^re  ^te  Diagnostics  Performing  as  Designed?  Test  Voids? 

Knowledge  Engineering  starts  providing  answers! 
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Why  Care  About  ID  Knowledge? 


ID  CLKS 


■  Mining  data  is  mining  knowledge 

-  Data  mining  utilizes  automated  search  algorithms  (patterns,  similarities, 
correlations  or  text  matching).  Data  results  are  visually  presented  to  the  user 
(better  understanding  and  improved  judgments). 

-  Knowledge  has  potential 

-  Properly  maintained 

■  Optimized  for  use  (IT  independent) 

-  Valued 

■  Trying  to  tell  us  something  -  are  we  listening? 

■  Look  into  the  crystal  ball  -  what  do  you  see? 

-  Categorized 

■  Impact  and  message 

-  Good,  missing,  dirty  or  bad  data 

-  Available  at  the  point  of  use  and  to  the  next  specialty 

■  Timely  and  meaningful  manner 

■  DATA? 
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Previous  Boeing  Knowledge  Study  Results 

ID  CLKS 


■  Aircraft  Program  Knowledge  Discovery  2001 

-  Discovered  correlations  between 
aircraft/system  events 

-  Identified  emerging  system  issues/trends 

-  Identified  cause  of  part/system  failure 


■Aircraft  Program  Data  Mining  Study  Results  2002 


-  Identified  aircraft  with  significant  failure  clustering 

-  Identified  recurrent  faults  with  specific  underlying  relationships  to 
aircraft  parametric  data 

-  Identified  separate  nuisance  fault  codes  for  consolidation 

-  Identified  ideas  of  improving  data  quality  for  wiring  faults 
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Knowledge  Wheel 
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Transactional  (Operational)  Databases 


ID  CLKS 


■  MANY  databases  used  for  day  to 
functions 

-  But  NOT  a  data  warehouse 


Transactional  Sources  of  Data 


ID  CLKS 


Fault  data 

•  Fault  code 

•  System  component 
performance 

•  Operational  context 
parameters 

•  Flight  data  recording 


Maintenance  data  Logistics  data 

•  Pilot  debrief  •  Aircraft/part  identification  and 

•  Procedures  used  configuration 

•  Actions  taken/parts  replaced  •  Aircraft  part  usage  (sorties, 

•  Time  for  action/personnel  hours) 

•  Subsystem/component  test  results  •  LRU/component  history 

•  Spares  disposition 


•  Flight  data  for  single  aircraft 
past  flights 

•  Specific  squadrons 

•  Bases/geographic  regions 

•  Correlation  to  specific  flight 
text 

•  Frequency  of  occurrence 


^Manufacturing  data 

•  Lot  number 
•  Acceptance  test  results 

•  Aircraft  configuration 
•  Component  lot  number 
•  Maintenance  actions 
•  Personnel  experience 
•  I  level  test  results 
•  Number/pattern  of  CNDs  and 
RTOKs 


Copyright© 2007  Boeing.  All  rights  reserved. 


8 

10/24/2007 


Data  Warehouse 


ID  CLKS 


■  Database  designed  to  support 

-  Extract,  Transform  and  Load  (ETL 
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Data  Mining  and  Aircraft  Failure 
Visualization 

ID  CLKS 


Exploration 

•Analysis 
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7  P’s  Visual  Knowledge  and  Results 
Opportunities 

ID  CLKS 


■  People 

-  Is  there  a  training  or  staff  issue  driving  the  poor  diagnostics,  Can- 
Not-Duplicate,  Bench-Check-OK,  etc? 

-  Are  required  entries  within  maintenance  system  filled  out 
completely  and  correctly? 

-  Is  there  an  opportunity  to  update  the  maintenance  system? 

■  Process 


-  Business  process  improvement?  Is  process  too  complicated,  not 
accessible? 


-  Could  a  LEAN  approach  provide  a  better  solution? 

■  Procedure 


-  Does  the  maintenance  procedure  need  updating  or  smart 
diagnostics  updated? 

-  Is  there  a  false  alarm  that  needs  masking? 

■  Portal 


-  One  web,  one  login,  common  user  interface? 
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7  P’s  Visual  Knowledge  and  Results 
Opportunities  (cont) 

ID  CLKS 


■  Prognostics 

■  If  A  and  B  are  bad,  then  C  will  fail  with  a  certain  period? 

■  Profit 

-  Is  the  knowledge  discovery  or  change  the  exception  or  the  rule? 

-  Too  costly? 

■  Possibilities 

-  What  is  the  data  and/or  the  metrics  trying  to  tell  us? 

-  Share  the  knowledge  with  subject  matter  experts  from  applicable  business 
disciplines.  Knowledge  drives  capturing  of  focused  domain  knowledge? 

-  If  a  wiring  repair  maintenance  action,  compare  job  closeout  WUC  with  text  mined 
closeout  narrative.  Flag  due  to  incorrect  WUC  assignment  (LRU  instead  of  wire 
repair).  Unnecessary  LRU  failures  which  drives  spares? 

-  If  relationships  between  flight  parameters,  generated  failures  and  human 
observables  exist,  consider  updating  diagnostics  accordingly? 
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Knowledge  Wheel  Disciplines 

ID  CLKS 


■  Knowledge  collaboration  of  related  information 
between  business  disciplines  improves  ID  influence 
and  maturity  including  quality  and  timeliness  of 
applicable  decision  making  processes 

-  Engineering 

-  Diagnosis  Development 

-  Technical  Publications 

-  Support  Equipment 

-  Maintenance  Tools 

-  Field  Service  Representatives 

-  Reliability  and  Maintainability 

-  Spares 

-  Training 

-  Metrics 

-  Production  Operations 


MTTR 
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Who  benefits? 


ID  CLKS 


Boeing 

-  Integrated  Diagnostic  Engineers 

■  Fast  &  accurate  fault  rule  implementation 

-  CFRS/IMIS/AME/SMART  TPS 

■  Consistent  ICD  and  rule  creation 

-  Spares  &  Provisioning 

■  Can  more  accurately  predict  what  and  when  parts  will  fail 

-  Reliability  &  maintainability 

■  Access  to  more  accurate  failure 

-  Field  Service  Reps 

■  Fleet  reports,  trends  and  proactive  information 

-  Training 

■  Focused  Curriculum  Updates 

Customer 

-  Aircraft  Pilots 

■  More  reliable  and  predictable  aircraft 

-  Maintainers  at  Various  Levels  of  Maintenance 


Test  y 
Results  / 

Provide  ICD 
&  Rules 

Provide  Data 
Analyze  Data 


■  View  of  what  to  expect  and  complete  aircraft  history 

■  Use  of  BIT  data  with  Automated  Test  Sets  (Directed  TPS) 
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Summary 


ID  CLKS 


■Challenge 

-  Implement  an  effective  method  of  ID  knowledge  use  and  integration 
across  specialties 

■  Provide  accurate  and  up-to-date  diagnostics 

■  Reduce  disruptive  maintenance  problems 

■  Reduce  cost  of  maintenance 

■  Aid  planning  for  support  of  future  missions 

■  Solution 

-  Develop  data  warehouse  and  utilize  data  mining 

-  Use  predictive  modeling  to  cluster  defects  and  define  influences 

-  Build  on  past  studies  and  lessons  learned 

■  Future  Benefits 

-  Enhanced  domain  knowledge  capture,  training  and  transfer 

-  Evaluated  hidden  relationships  and  cost  saving  opportunities 

-  Increased  smart  diagnostics  maturation  and  decreased  false  alarms 

-  Knowledge  builds  upon  knowledge 
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Knowledge  is  Power  - 

When  properly  Engineered 


ID  CLKS 


1 0/24/2007 
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A  Strategy  for  Improved  System 

Assurance 

October  24,  2007 


Kristen  Baldwin 

Deputy  Director, 

Software  Engineering  and  System  Assurance 
Office  of  the  Under  Secretary  of  Defense 
Acquisition,  Technology  and  Logistics 


i 


Assurance  Efforts  Update 


•  Defense  Industrial  Base  Information  Assurance  Policy  Team 
Efforts 


•  System  Assurance  Working  Group  Efforts 

-  Current  Tasking 

-  6-bar  construct 

-  Progress 

•  System  Assurance  Guidebook 

-  Intent 

-  Current  Status  and  Way  Ahead 

•  Program  Protection  Policy 

•  Software  Assurance  Initiative 

-  Software  Engineering  Institute 

•  Overall  Systems  Assurance  Progress  Report 
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System  Assurance 


•  We  continue  to  be  concerned  with  assurance  of  our 
critical  DoD  assets: 

•  Critical  information 

•  Critical  technologies 

•  Critical  systems 

•  Observations: 

-  Increasing  numbers  of  network  attacks  (internal  and  external  to  DoD) 

-  Broader  attack  space 

•  Trends  that  exacerbate  our  concerns: 

-  Globalization  of  our  contracts,  expanding  the  number  of  international 
participants  in  our  system  developments 

-  Complex  contracting  arrangements  that  further  decrease 
transparency  below  prime,  and  visibility  into  individual  components 


These  trends  increase  the  opportunity  for  access  to  our  critical 

assets,  and  for  tampering 


Matrixed  participation  by  DoD  and  DIB  representatives  in  all  5  Action  Teams 
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System  Assurance  Working  Group  Update: 

6-bar  approach 


Requirements 


Acquisition 


Contracting 


Development 


Oversight 


.Compliance 


•  “Holistic”  approach,  end-to-end  spectrum  to  capture  the  most 
stakeholders 

-  Note:  Intelligence  Stakeholder  is  embedded  in  and  across  all  “bars” 

•  Concentrate  on  six  areas  of  interest,  which  also  happen  to  be  logical 
grouping  of  discipline  interest  and  existing  policies 

•  Within  each  “bar”,  identify  processes,  policies  to  leverage  for  system 
assurance 
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Assurance  Definitions 


By3'iB/nsj  A 


u/cince  JffjpJsjffjemicWOf]  Siriiisgy 


•  Joint  Publications  &  Definitions 

JCI DS  *  ICD>  CDD,  CPD  boiler  plate 

•  NR  KPP  Attributes 

(existing  process) 
(existing  process) 
(existing  process) 

Acquisition 

•  “Cost  of  Doing  Business” 

•  Guidance  for  SEP,  ISP,  TEMP 

•  CPI  pre-work  in  FAA  and  Functional  Needs  Analysis  (FNA) 

•  Combine  Plans  where  possible  (e.g.,  OPSECoPPP) 

•  Threat  and  risk  collaboration 

(existing  process) 
(existing  process) 
(existing  process) 
(existing  process) 
(existing  process) 

Six  task  blocks 
shown  for  breakout 
purposes  only. 

Contracting 

•  Terms  &  Conditions 

•  Standard  Contract  Language 

•  SDP,  PPP 

(existing  process) 
(existing  process) 
(existing  process) 

Not  aligned  with  timeline,  yet 

Development 

•  Technical  Requirements 

•  Mitigation  Guidance  (Nil,  DSS) 

•  Software  tools  (CPI-ID’d,  Assess) 

(existing  process) 
(existing  process) 
(existing  process) 

Oversight 

•  SEP  and  SETR 

•  PM  Checklist 

•  MDA,  Milestone,  PSR  tools 

(existing/new) 
(existing  process) 
(existing  process) 

Compliance 

•  Acquisition  Integrity  Office  (existing  process) 

•  DCMA  (existing  process) 

•  CIS  (existing  process) 

JROC 


DAB  DSAB  DAB  DSAB  JROC  DAB  ITAB 

ITAB  ITAB 


JROC  DAB  DSAB 

ITAB 


DoD  Requirements  Development  Process 


A Intelligence  Updates  Required 


Acquisition  Path  Forward 


•  Create  a  ‘framework’  to  integrate  multiple  security  disciplines 
and  policies 

-  Leverage  5200.39:  expand  CPI  definition  to  include  system 


assurance  and  total  life  cycle 

•  Use  the  Program  Protection  Plan  (PPP)  to  identify  CPI  and 
address  assurance  for  the  program 

-  Link  plans  (e.g.,  Anti-Tamper,  Software  Protection,  System 
Engineering,  Assurance  Case) 

•  Modify  Acquisition  and  System  Engineering  guidance  to 
integrate  system  assurance  across  the  lifecycle 

-  Milestone  Decision  Authority  visibility 

-  Guidebook  on  Engineering  for  Assurance  for  program 
managers/engineers 


Raise  the  bar: 

Awareness 

-  Knowledge  of  the  supply  chain 

-  Who  has  access  to  our  critical  assets 

Protection 

-  Protect  critical  assets  through  security  practices  7 

-  Engineer  our  systems  for  assurance 

Current  Systems  Security  Policies 


Component  Protection  Sought 


Defense- 

In-Depth 

Intelligence 

Supply  Chain 

Engineering 

Certification 

Documented  Plan 


Critical 

Functionality 

Non-  Security 

Security 


Critical 

Information 

Classified  Un¬ 
classified 


Critical 

Technology 

Software  Hardware/ 
Firmware 


SA 


5200.39 


TF 


.GCZNIAP 


FIPS 


ISPr 


NISP 


5200.39 


.GCZNIAP 


FIPS 


DIACAP 


IA 


SPI 


OPSEC 


DIACAP 


Anti- 

Tamper 


5200.39 


Policy  Ownership 

DoD  -  CIO/DSS 

DoD  -  AT&L 

DoD  -  AT&L/S&T 

DoD  -  CIO/DISA 

CC/NSA 

DoD  -  NSA 

DoD  -  USD(I) 

NIST 
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Proposed  Framework  for  Security  Policies 


Defense- 

In-Depth 

Intelligence 

Supply  Chain 

Engineering 

Certification 

Documented  Plan 


Component  Protection  Sought 

Critical  Critical  Critical 

Functionality  Information  Technology 

Security  Classified 


Non- 

Security 


Un¬ 

classified 


Software 


Hardware/ 

Firmware 


5000.1  /.2/Systems  Engineering 


Proposed  Framework  with  5200.39 


IA 

CC/NIAP 

FIPS 

DIACAP 

OPSEC 

IA 


SPI 


DIACAP 


Anti- 

Tamper 


Policy  Ownership 

DoD  -  CIO/DSS 

DoD  -  AT&L 

DoD  -  AT&L/S&T 

DoD  -  CIO/DISA 

CC/NSA 

DoD  -  NSA 

DoD  -  USD(I) 

NIST 
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Critical  Program  Information 


New  Definition  -  Draft  DoDI  5200.39: 


•  E3.6.  Critical  Program  Information  (CPI).  Elements  or  components  of 
an  RDA  program  that  if  compromised,  could  cause  significant 
degradation  in  mission  effectiveness,  shorten  the  expected  combat- 
effective  life  of  the  system,  reduce  technological  overmatch, 
significantly  alter  program  direction,  or  enable  an  adversary  to 
counter,  copy,  or  reverse  engineer  the  technology  or  capability. 

•  E3.6.1.  Technologies  become  eligible  for  CPI  selection  when  a  DoD 
Agency  or  military  component  invests  resources  to  demonstrate  an 
application  for  the  technology  in  an  operational  setting,  or  in  support 
of  a  transition  agreement  with  a  Program  Manager. 

•  E3.6.2.  Includes  information  about  applications,  capabilities, 
processes,  and  end-items. 

•  E3.6.3.  Includes  elements  or  components  critical  to  a  military  system 
or  network  mission  effectiveness. 
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Notional  Assurance  Implementation 


Identify  CPI 

Identify  threats 

Develop  Plans  (SEP,  TES) 


Approved  SEP,  TEMP  with 
details  on  Assurance 
Milestone  Decision  approves 
plans,  sets  SDD  criteria 


•  Conduct  PSRs  for  Program 
•Protection  Plans 


Concept 

Technology 

Refinement 

Development 

\  Concept 

Y  Decision 

Sustainment  security  plans  in  place 
Maintenance  providers  meet  security 
practice 

Upgraded  HW/SW  configuration 
managed,  validated  and  verified 


(Program 

Initiation) 


/c\ 


IOC 


FOC 


System  Development 
&  Demonstration 


0 


CDR 


Production  & 
Deployment 


LRIP/IOT&E 


0 


FRP 

Decision 

Review 


Operations  & 
Support 


•  Source  selection  consideration 

of  supplier  FOCI  and  security  practices 

•  Technology  Readiness  Assessment 

•  CPI  entered  in  Horizontal  Protection 
•Database 


Write  Program  Protection  Plan  (PPP) 
Designs  meet  assurance  plans 
Initial  verification  and  validation  of  critical 
components 


Total  Lifecycle  Approach  to  Assured  Systems 
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Program  Protection  Plans 


•  Policy 

-  Revised  DoD  5200.39  policy 

-  DoD  5000.2  -  Deliverable  at  MS  B 

•  Guidance 

•  DAG  Chapter  4  and  8,  modified  to  reflect  policy  changes 

•  NDIA  System  Assurance  Guidebook 

•  Revised  SEP  and  TEMP  Guides 

•  Support 

-  Develop  on-site  Training 

•  Defining  CPI  consistent  with  new  version  of  DODI  5200.39 

•  Protecting  CPI  and  documenting  protection  in  PPP 

-  Senior  level  support  provided  to  assist  programs  in  defining, 
implementing,  and  documenting  protection  of  CPI  in  PPP 
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Development  Path  Forward 
SA  Guidebook 


•  Augments  system  engineering  from  documentation  through 
engineering  processes  and  technical  reviews 

-  Introduced  as  early  as  possible  -  Where  there  is  the  greatest  impact 

-  Continue  through  the  life  cycle 

•  Consistent  with  international  standard  and  current  best  practices 

-  E.g.,  Guidebook  approach,  presentation  of  process  /  procedure  consistent 
with  ISO/IEC  15288  standard  for  System  Engineering 

-  Integrates  consideration  and  leverages  numerous  existing  program 
protection  or  security  disciplines  (e.g.,  IA,  AT,  SwA,  SPI,  PPP) 

-  Existing  information  security  /  assurance  material  is  summarized,  and 
leveraged  by  reference,  not  repeated 

•  Enhanced  vulnerability  detection  techniques 

•  SwA  Body  of  Knowledge 

•  Intent  is  to  provide  practical  guidance  on  augmenting  systems 
engineering  practice  for  system  assurance 

-  Defines  “Engineering-in-Depth”! 
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■ 

Best 

Practice 

Standards 

Instructions 

Directives 

NIST,  NSA 
Guidance 

Etc. 


Systems  Assurance  Guidebook 

. , . 

T 

. 

Systems  Engineering  View 

I 

ISSE/IA  View 

# 

Program  Management  View 

Handbook 


■\ 


Others  as 
needed... 


Future:  Link  to  Acquisition  Guidance,  Evolve/Implement  into  training,  education 
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Cliff  Notes 


Guidebook  Construct 


Table  Of  Contents 

-  1 .  Introduction  and  Organization 

•  Definition  of  System  Assurance 

•  1.1  Scope 

•  1 .2.  Purpose 

•  1 .3  Audiences  and  Applications 

•  1 .4  Related  Disciplines 

•  1 .5  Relationships  of  Policies,  Standards  and  Efforts 

•  1 .6  Organization  of  Document 

-  2.  Context  of  Systems  Assurance 

-  3.  Guidance  (mapped  to  ISO/IEC  15288 

•  3.1  Agreement  Process  (ISO/IEC  15288  section  5.2) 

•  3.2  Enterprise  Process  (ISO/IEC  15288  section  5.3) 

•  3.3  Project  Processes  (ISO/IEC  15288  section  5.4.1) 

•  3.4  Technical  Processes  (ISO/IEC  15288  section  5.5) 

-  4.  Examples 

•  4.1  Guidebook  Implementation  Examples 

•  4.2  Assurance  Case  Development  Example 

-  5.  Documentation  Examples 

-  6.  Glossary  &  Acronyms 

-  7.  Bibliography 


Contact  us  to  participate  in  stakeholder  review 


Guidebook  Construct  con*t 


•  Table  Of  Contents 

-  Additional  Material 

•  Section  A:  Systems  Assurance  Concept  and  Methodology 

•  Section  B:  Correspondence  with  Existing  Documentation, 

Standards  efforts,  etc. 

•  Section  C:  Contacts  in  Communities  of  Interest  and 

Practice 

•  Section  D:  Anti-Tamper 

•  Section  E:  Enterprise  Processes 

•  Section  F:  Technical  Guidance  Research  &  Development 

(R&D) 

•  Index 


Contact  us  to  participate  in  stakeholder  review 
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Guidebook  Status 


•  Stakeholder  review  -  Comments  due  31  Oct  07 

-  Request  copy  for  comment  ATL-SS A@osd .mil 

•  Comment  adjudication  and  release  by  31  Dec  07 

-  Version  0.9  of  the  Guidebook,  to  be  updated  over  time 

•  Pilots 

-  Systems  Assurance  innovators  and  areas  where  comprehensive 
expertise  in  one  or  more  relevant  domains  exists 

-  Starting  Summer,  2007 

•  Write  specific  stakeholder  views 

-  Focus:  Derived  from  the  Guidebook,  “get  the  right  content”  (by 
audience) 


Contact  us  to  participate  in  stakeholder  review 


System  Assurance  Overall  Progress  Report 


UNCLASSIFIED 
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System  Assurance  Progress  Report 
-a  sampling  of  activities 


Requirements  -  JCIDS 

-  Modify  Joint  Publications  &  Definitions  to  include  SA 

-  Modify  ICD,  CDD,  CPD  boiler  plate  to  incorporate  SA 

-  CPI  pre-work  in  FAA  and  Functional  Needs  Analysis  (FNA) 

-  Modify  NR  KPP  Attributes  to  address  SA 

-  Develop  text  to  discuss  Systems  Assurance  within  JCIDS  documents 
\E\  Sample  boiler  plate  presented  at  SAWG  meeting  -  7  June  2007 


Acquisition  -  Program  Protection  Planning  (PPP)  Process 

-  Define  process  required  to  identify  CPI  components 

\E1  Submitted  edits  to  DODI  5200.39  with  definition  of  CPI  to  incorporate 
SA  interests  -  May  2007 

\E\  Drafted  formal  PPP  review  process  slide  set  -  18  May  2007 
\E\  Conducted  review  of  Component  PPP  processes,  tools  -  1  Aug  2007 
m  Developed  and  submitted  PPP  process  resource  estimate  -  30  Aug  07 
□  Draft  common  PPP  development  process  -  due  October  2007 
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System  Assurance  Progress  Report 


Development  -  Guidance  for  SEP,  ISP,  TEMP 

E  Updated  Systems  Engineering  Plan  (SEP)  Guide  to  include  system 
assurance  -  Aug  2007 

E  Modified  Defense  Acquisition  Guide  (DAG)  Chapter  4  on  systems 
engineering 

□  Modifying  DAG  Chapter  8 

Development  -  Guidebook 

E  NDIA  Guidebook  released  to  stakeholders  -  19  Sep  2007 

□  Adjudicate  comments  and  release  Version  0.9  -  31  Dec  2007 

Oversight  -  SA  Content  for  Program  Support  Reviews 

-  Define  how  programs  should  be  assessed  for  compliance  with  systems 
assurance  policy  and  guidance 

E  Developed  guidance  and  questions  -  May  2007 

E  Conducted  pilot  assessment  -  June  2007 

Intelligence  Community  collaboration 

E  Developed  and  submitted  estimate  of  impact  on  Cl  resources  to  conduct 
threat  assessments-  May  2007  20 


System  Assurance: 

What  does  success  look  like? 


The  requirement  for  assurance  is  allocated 
among  the  right  systems  and  their  critical 
components 

DoD  understands  its  supply  chain  risks 

DoD  systems  are  designed  and  sustained 
at  a  known  level  of  assurance 

Commercial  sector  shares  ownership  and 
builds  assured  products 

Technology  investment  transforms  the 
ability  to  detect  and  mitigate  system 
vulnerabilities 


Prioritization 


Supplier 

Assurance 


Engineering- 
In-Depth 

Industry 
Outreach 


Technology 

Investment 


Questions/Comments 


22 

UNCLASSIFIED 


DoD  Security  Policies 


•  The  DoD  Acquisition  System  must  develop  secure  weapon 
systems  and  must  increase  the  security  of  the  acquisition 
process  itself. 

•  The  purpose  of  secure  warfighting  systems  and  acquisition 
processes  is  to  protect  the  DoD  technology  lead,  develop 
warfighting  systems  that  cannot  be  usurped  or  disabled,  and 
ensure  the  secure  flow  of  information  during  war  and 
peacetime  for  its  warfighting  systems  and  corporate 
infrastructure. 

•  Primary  policy  concerned  with  securing  the  warfighting 
acquisition  process  and  systems: 

•  DODI  5200.39  Security,  Intelligence,  and  Counterintelligence  Support  to 
Acquisition  Program  Protection 
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DoD  Security  Policies 


•  Countermeasures  -  methods  for  protecting  CPI 

-  System  Assurance  (DAG  Chapter  4  &  8,  MIL-HDBK-1985  Secure 
System  Design) 

-  Classification  (DODD  5200.1  Information  Security  Program,  ISP) 

-  Network  security  (DOD8500.01  E  Information  Assurance) 

-  Secure  communications  (C-5200.5  Communications  Security) 

-  Hardcopy  document  markings 

-  Physical  security  (DODI  5200.08  Security  of  DoD  Installations  and 
Resources) 

-  Operational  security  (DODD  5205.02  OPSEC) 
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Backup  Slides 
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Top  Software  Issues* 


.  The  impact  of  requirements  upon  software  is  not  consistently 
quantified  and  managed  in  development  or  sustainment. 


2.  Fundamental  system  engineering  decisions  are  made  without 
full  participation  of  software  engineering. 

3.  Software  life-cycle  planning  and  management  by  acquirers  and 
suppliers  is  ineffective. 

4.  The  quantity  and  quality  of  software  engineering  expertise  is 
insufficient  to  meet  the  demands  of  government  ana  the  defense 
industry. 

5.  Traditional  software  verification  techniques  are  costly  and 

ineffective  for  dealing  with  the  scale  and  complexity  of  modern 
systems. _ 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure 
execution  of  complex  software  in  distributed  environments. 

7.  Inadequate  attention  is  given  to  total  lifecycle  issues  for 
COTS/NDI  impacts  on  lifecycle  cost  and  risk. 

*NDIA  Top  Software  Issues  Worktop 
August  2006 


Fragmented  Systems  Security  Policies 


Each  policy: 

•  Affects  different  parts  of  the  life 
cycle 

-  R&D,  acquisition,  foreign  ownership 

•  Applies  to  a  different  subset  of  DoD 
systems 

-  NSS,  IT,  MDA,  ACAT  1 C,  etc. 

•  Assures  different  ‘type’  of 
components 

-  information,  leading  technology, 
functionality 

•  Mandates  a  different  set  of  defense 
tactics 

-  intelligence,  engineering,  documented 
plan,  certification  &  accreditation 


CC  -  Common  Criteria 

DIACAP  -  DoD  Certification  & 
Accreditation 

FIPS  -  Federal  Information  Processing 
Standards 

ITAR  -  International  Traffic  in  Arms 
Regulation 

IA  -  Information  Assurance 

ISP  -  Information  Security  Program 

NIAP  -  National  Information  Assurance 
Partnership 

NISP  -  National  Industrial  Security 
Program 

OPSEC  -  Operational  Security 

5200.39  -  DODD  5200.39  Security, 
Intelligence,  and  Counterintelligence 
Support  to  Acquisition  Program 
Protection 

SA  -  System  Assurance 

SPI  -  Software  Protection  Initiative 

TF  -  Trusted  Foundry 


Current  approach  does  not  have  systems-of-systems  perspective 
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System  Assurance  Context  for  the  PM 


System  Assurance  -  Working  Definition 
Level  of  confidence  that  a  system  functions  as  intended,  is  free  of 
exploitable  vulnerabilities,  and  protects  critical  program  information 
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Consequences  of  Fragmented  Systems 

Assurance  Initiatives 


•  Lack  of  Coherent  Direction  for  PMs,  and  others  acquiring 
systems 

-  Numerous,  uncoordinated  initiatives 

-  Multiple  constraints  for  PMs,  sometimes  conflicting 

-  Loss  of  time  and  money  and  lack  of  focus  on  applying  the  most 
appropriate  engineering  for  systems  assurance  for  each  system 

•  Synergy  of  Policy  -  Multiple  ownership 

-  Failure  to  capitalize  on  common  methods,  instruction  among 
initiatives 

•  DoD  Risk  Exposure 

-  Lack  of  total  life  cycle  view 

-  Lack  of  a  focal  point  to  endorse  system  assurance,  resolve 
issues,  advocate  PM  attention 

-  Lack  of  system-of-systems,  architecture  perspective  on  system 
assurance 

-  Potential  for  gaps  in  systems  assurance  protection 


Safety  of  Unmanned  Systems 


Sponsored  by 

Safety  Oversight  Council 
uisition-and  Technology 
Programs  Task  Force 
(DSOC  ATP  TF) 


Michael  H.  Demmick 
24  October  2007 


■  v*  t 


Agenda 


•  Leadership 

•  Background 

•  Objectives 

•  Approach 

•  Progress 

•  Organization 

•  Workgroup  participants 

•  Precepts  Review 

•  Final  Product 

•  Summary 


Unmanned  Systems 
_ Leadership _ 

•  OSD  Sponsor 

-  Mr.  Mark  Schaeffer,  Director, 
Systems  and  Software  Engineering 
&  Chairman,  DSOC  ATP  TF 

-  Dr.  Liz  Rodriquez-Johnson, 
Executive  Secretary,  DSOC  ATP  TF 


Why  Safety  of  UMSs? 
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UAV  launch  from  MDARS 


Talon  Swords 


Raytheon  UCAV 
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UMS  Level  of  Awareness  vs. 
Levels  of  Control 
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Background 


•  In  FY05,  the  OSD  Joint  Robotics 
Program  Coordinator  for  ground 
systems  tasked  Navy  to: 

-  Provide  unifying  safety  guidance  across  all 
ground  robotic  projects 

-  Establish  initial  safety  precepts  for  ground 
robotic  systems 

*  Program  Safety  Guidance 

*  Operational  Guidance 

*  System  Design  Safety  Guidance 

•  Results  briefed  at  2005  ISSC 
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Background 


•  October  2005  briefed  to  OSD  (DSOC  ATP  TF) 

•  ATP  TF  directed  expansion  of  effort  to  include 
all  Unmanned  Systems  (air,  ground,  and  sea) 

•  Emphasized  necessity  of  community  input 

-  Program  Management 

-  Design 

-  Test 

-  Operational 

-  Safety 

•  Emphasized  guidance  vice  direction 


10 


.  \^ri  •  b£p 


UMS  Safety  Objectives 


•  Focus  the  technical  community  on  the 
System  Safety  needs  for  UMS 

•  Specifically: 

1 .  Understand  the  safety  implications,  including 
legal  issues,  associated  with  the  rapid 
development  and  use  of  a  diverse  family  of 
unmanned  systems  both  within,  and  external  to, 
the  DoD. 

2.  Establish  and  agree  upon  a  standardized  set  of 
safety  precepts  to  guide  the  design,  operation, 
and  programmatic  oversight  of  all  unmanned 
systems. 

3.  Develop  safety  guidance,  such  as  design 
features,  hazard  controls  and  mitigators,  for  the 
design,  development,  and  acquisition  of 
unmanned  systems. 


Approach 

^  Involve  technical  community 

-  Six  Workgroups 

-  Approximately  80  technical  experts 

-  Government,  Industry,  Academia 

Maximize  Community  Awareness 

-  March  2006  Workshop 

•  300  attendees 

-  International  Systems  Safety  Conference  (ISSC) 

-  Association  of  Unmanned  Vehicles  International  (AUVSI) 

-  NDIA  Systems  Engineering  Conference 

Obtain  Feedback 

-  Web  Page  (http://www.ih.navy.mil/unmannedsystems) 

-  Tech  Panels  &  Reviews 

s  ISSC  (31  July  ■  4  Aug  2006) 

S  AUVSI  (29  -  31  Aug  2006) 

^  NDIA  Systems  Engineering  (23  -  26  Oct  2006) 

S  Mr.  Schaeffer’s  Systems  Engineering  Forum 
NDIA  Systems  Engineering  (22  -  25  Oct  2007) 


Road  to  Completion 


^ Held  Three  Workshops 

-  March  2006,  Huntsville 

-  May  2006,  Crystal  City 

-  June  2006,  Crystal  City 

*  Developed  Safety  Precepts 

-  Programmatic  safety  precepts  (6) 

-  Operational  safety  precepts  (5) 

-  Design  safety  precepts  (19) 

'S Developed  more  detailed  design  safety  “best 
practices”  (safety  precept  clarification  tables) 
(ongoing) 

^ USD  (AT&L)  issued  the  Guide  on  17  July  2007 


xM-PIa 


Workshop  Organization 


'S  Six  Workgroups 

1 .  Precept  Development 

2.  Weapons  Control 

3.  Situational  Awareness 

•  Human-Machine  Interface 

•  Machine-Machine  Interface 

4.  Command  and  Control 

5.  States  and  Modes 

6.  Definitions/Common  Taxonomy 


Unmanned  Systems 
Management  Team 


•  Members 

-  Mr.  Dave  Schulte 

-  Mr.  Ed  Kratovil 

-  Mr.  Jim  Gerber 

-  Ms.  Rhonda  Barnes 

-  Mr.  Danny  Brunson 

-  Mr.  Josh  McNeil 

-  Mr.  Bill  Pottratz 

-  Dr.  Tom  English 

-  Mr.  Steve  Mattern 

-  Mr.  John  Canning 

-  Mr.  Bob  Schmedake 
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Workgroup  Participants 


Precepts: 

Mr.  Josh  McNeil  (Army) 

-  Mr.  Woody  Eischens  (OSD) 

-  Mr.  Clif  Ericson  (EG&G) 

-  Mr.  Tom  Garrett  (Navy) 

-  Mr.  Hui-min  Huang  (NIST) 

-  Mr.  Bob  Jacob  (Navy) 

-  Mr.  Mike  Logan  (NASA) 

-  Mr.  Ranjit  Mann  (APT) 

-  Mr.  Jack  Marett  (Westar) 

-  Mr.  Charles  Muniak  (LMCO) 

-  Ms.  Kristen  Norris  (AOT) 

-  Mr.  Alan  Owens  (Air  Force) 

-  Mr.  Scott  Rideout  (USMC) 

-  Ms.  Peggy  Rogers  (Navy) 

-  Mr.  Craig  Schilder  (APS) 

-  Mr.  Arthur  Tucker  (SAIC) 

-  Mr.  Frank  Zalegowski  (Navy) 

-  Mr.  Jim  Zidzik  (Navy) 

-  Mr.  Don  Zrebieck  (Navy) 


Weapons  Control: 

Mr.  Bill  Pottratz  (Army) 

-  Mr.  Scott  Allred  (USMC) 

-  Mr.  Bill  Blake  (ATK) 

-  Dr.  Craig  Bredin  (Westar) 

-  Ms.  Mary  Ellen  Caro 
(Navy) 

-  Mr.  John  Deep  (USAF) 

-  Mr.  Jon  Derickson  (BAE) 

-  Mr.  John  Filo  (Navy) 
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Special  Thanks 
“Heavy  Lifters” 


Mr.  Jim  Gerber 
Mr.  Mike  Demmick 
S Mr.  Josh  McNeil 
v Ms.  Rhonda  Barnes 
▼  Mr.  Danny  Brunson 
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UMS  Safety  Precept  Definitions 


Programmatic  Safety  Precept  (PSP)  =  Program 
management  principles  &  guidance  that  will  help  ensure 
safety  is  adequately  addressed  throughout  the  lifecycle 
process.  (6) 

Operational  Safety  Precept  (OSP)  =  a  safety  precept 
directed  specifically  at  system  operation.  Operational  rules 
that  must  be  adhered  to  during  system  operation.  These 
safety  precepts  may  generate  the  need  for  Design  Safety 
Precepts.  (5) 

Design  Safety  Precept  (DSP)  =  General  design 
guidance  intended  to  facilitate  safety  of  the  system  and 
minimize  hazards.  Safety  design  precepts  are  intended  to 
influence,  but  not  dictate,  specific  design  solutions.  (19) 


Safety  Precepts  for  UMS 


OSD  Policy 


PM/Operators/ 
User  reps 


PM/Industry 
Design  Team  “S 


Provide  PMs,  designers,  and  systems  safety  managers  with  appropriate  safety 
guidelines  and  best  practices,  while  maintaining  PM’s  flexibility 


Safety  Design  Guidelines 


Unmanned  Systems 
Safety  Design 


Manned  Systems 
Safety  Design  “Best 
Practices” 

-MILSTDS 

- STANAGS 

-Handbook: 


Common 
To  Both 


Best  Practices 


Are  we  creating  two  sets  of  safety  criteria: 
one  for  manned  systems,  and  one  for  unmanned  systems?? 


Unique  to 
Manned  System 


Unique  to 
Unmanned  System 


Creating  another  set  of  safety  requirements?  No 


.  \^ri  •  b£p 


Safety  Precepts 


^  Did  not  previously  exist 

^  Evolved  through  an  arduous,  but 
thorough,  systems  engineering 
process  over  the  past  2  years 

^  Separate  study  was  performed  to 
determine  if  current  DoD  and/or 
Service-specific  policies  addressed 
each  of  the  safety  precepts 


Safety  Precepts  (cont’d) 


The  results  of  this  study  indicate: 

s  Safety  precept  PSP-1  is  completely  addressed  in  both  DoD  and 
Service-specific  policies. 

s  Three  precepts  (PSP-4,  PSP-6,  and  DSP-1)  are  completely 
addressed  in  DoD  policy  and  are  partially  addressed  in  Service- 
specific  policies. 

s  Four  precepts  (PSP-3,  DSP-11,  DSP-12,  and  DSP-19)  are  partially 
addressed  in  both  DoD  and  Service-specific  policies. 

✓  Nine  precepts  (PSP-2,  OSP-1,  OSP-3,  OSP-5,  DSP-7,  DSP-13, 
DSP-14,  DSP-16,  DSP-18)  are  not  addressed  in  DoD  policy  but 
are  partially  addressed  in  Service-specific  policy. 

S  Twelve  precepts  (PSP-5,  OSP-2,  OSP-4,  DSP-2,  DSP-4,  DSP-5, 
DSP-6,  DSP-8,  DSP-9,  DSP-10,  DSP-15  and  DSP-17)  are  not 
addressed  in  DoD  nor  Service-specific  policies. 

s  One  precept  DSP-3  was  not  mapped  to  policy. 


Final  Product 

UNMANNED  SYSTEMS  SAFETY  GUIDE  FOR  DOD 

ACQUISITION 
27  June  2007 

^  Document  contains  descriptive  and 
clarifying  text  for  each  precept. 

v  Includes  definitions 

But, ...comments/lessons  learned  are 
still  requested  for  future  updates 

-  NOSSA  Website 

(http://www.ih.navy.mil/unmannedsystems) 


USD  (AT&L)  UMS  Memorandum 


THE  UNDER  SECRETARY  OF  DEFENSE 

3010  DEFENSE  PENTAGON 
WASHINGTON.  DC  £0301  3010 


JUL  1  7  Z0D7 

ACDUl*1TK)H, 

TECHNOLOGY 
AN O  LOGISTICS 


MEMORANDUM  I -’OR  SECRETARIES  OF  THE  MILITARY  DEPARTMENTS 
CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 
UNDER  SECRETARIES  OF  DEFENSE 
COMMANDERS  OF  THE  COMBATANT  COMMANDS 
ASSISTANT  SECRETARY  OF  DEFENSE  (NETWORKS  & 
INFORMATION  INTEGRATION) 

DIRECTOR,  DEFENSE  RESEARCH  AND  ENGINEERING 
DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
DIRECTOR,  PROGRAM  ANALYSIS  AND  EVALUATION 
DIRECTORS  OF  TI  IE  DEFENSE  AGENCIES 

SUBJECT:  Unmanned  Systems  Safely  Guidance 

In  March  2006,  the  Defense  Safety  Oversight  Council  Acquisition  and  Technology 
Programs  Task  Force  (A TP  TF)  initiated  a  study  to  identify  Lhc  unique  safety  challenges 
of  unmanned  systems  (UMSs),  especially  those  systems  carry  ing  and  deploying  weapons 
in  a  joint  environment.  These  safety  challenges  significantly  increase  as  more  UMSs  are 
fielded  and  used  in  the  same  warfighting  environment. 


Using  a  collaborative  process  with  experienced  personnel  from  all  Services,  the 
ATP  TF  developed  the  “Unmanned  Systems  Safety  Guide  for  DoD  Acquisition”  ^  A 
provide  programmatic,  operational,  and  design  guidelines  to  support  the  developmjjTand 
fielding  of  safe  UMSs.  Please  you  use  the  Guide,  found  at  http://www.acu.osd.mil/atptl7, 
to  help  identify  and  mitigate  hazards  and  their  associated  risks  for  all  UMS  types. 


For  those  UMSs  that  are  AC  AT  ID  program,  the  UMS  safety  gu  _ 
special  interest  item  during  OSD  Program  Support  Reviews.  UMS-specitlc  j_ 
have  been  added  to  the  Defense  Acquisition  Program  Support  methodology  to  guide  the 
evaluation  of  how  successfully  programs  have  engineered  UMSs  to  reduce  safety  risks  to 
acceptable  levels. 


use  the  Guide  to  help 
identify  and  mitigate  hazards 
and  their  associated  risks  for 
all  UMS  types.” 


“For  those  UMSs  that  are 
ACAT  ID  Programs,  the  UMS 
safety  guidelines  will  be  a 
special  interest  item  during 
OSD  Program  Support 
Reviews.” 
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Programmatic  Safety  Precepts 


PSP-1*:  The  Program  Office  shall  establish  and  maintain  a  system 
safety  program  (SSP)  consistent  with  MIL-STD-882. 

PSP-2*:  The  Program  Office  shall  establish  unifying  safety 
precepts  and  processes  for  all  programs  under  their 
cognizance  to  ensure: 

-  Safety  consistent  with  mission  requirements,  cost  and 
schedule 

-  Mishap  risk  is  identified,  mitigated  and  accepted. 

-  Each  system  can  be  safely  used  in  a  combined  and 
joint  environment 

-  That  all  safety  regulations,  laws,  and  requirements  are 
met. 

PSP-3*:  The  Program  Office  shall  ensure  that  off-the-shelf  items 
(e.g.,  COTS,  GOTS,  NDI),  re-use  items,  original  use  items, 
design  changes,  technology  refresh,  and  technology 
upgrades  (hardware  and  software)  are  assessed  for 
safety,  within  the  system. 


Programmatic  Safety  Precepts 

(Cont’d) 


PSP-4*:  The  Program  Office  shall  ensure  that  safety  is 
addressed  for  all  life  cycle  phases. 

PSP-5:  Compliance  to  and  deviation  from  the  safety  precepts 
shall  be  addressed  during  all  Milestone 
decisions  and  formal  design  reviews  such  as  System 
Requirements  Review  (SRR),  Preliminary  Design 
Review  (PDR),  and  Critical  Design  Review  (CDR). 

PSP-6*:  The  Program  Office  shall  ensure  UMS  designs  comply 
with  current  safety  and  performance  criteria. 


Note:  While  the  document  serves  only  as  a  guide,  usage  of  the  terms 
“shall”  and  “should”  reflects  the  level  of  concern  of  the  safety 
community 

*  Denotes  applicability  to  both  manned  and  unmanned  systems. 


Operational  Safety  Precepts 


OSP-1 :  The  controlling  entity(ies)  of  the  UMS  should  have 
adequate  mission  information  to  support  safe 
operations. 

OPS-2:  The  UMS  shall  be  considered  unsafe  until  a  safe  state 
can  be  verified. 

OPS-3:  The  authorized  entity(ies)  of  the  UMS  shall  verify  the 
state  of  the  UMS,  to  ensure  a  safe  state  prior  to 
performing  any  operations  or  tasks. 

OSP-4*:  The  UMS  weapons  should  be  loaded  and/or  energized 
as  late  as  possible  in  the  operational  sequence. 

OSP-5*:  Only  authorized,  qualified  and  trained  personnel,  with 

the  commensurate  skills  and  expertise  using  authorized 
procedures,  shall  operate  or  maintain  the  UMS. 
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Design  Safety  Precepts 


DSP-1*:  The  UMS  shall  be  designed  to  minimize  the  mishap  risk  during 
all  life  cycles  phases. 

DSP-2:  The  UMS  shall  be  designed  to  only  respond  to  fulfill  valid 
commands  from  the  authorized  entity(s). 

DSP-3:  The  UMS  shall  be  designed  to  provide  information,  intelligence, 
and  method  of  control  (I2C)  to  support  safe  operations. 

DSP-4*:  The  UMS  shall  be  designed  to  isolate  power  until  as  late  in  the 
operational  sequence  as  practical  from  items  such  as:  a) 
Weapons,  b)  Rocket  motor  initiation  circuits,  c)  Bomb  release 
racks,  or  d)  Propulsion  systems. 

DSP-5*:  The  UMS  shall  be  designed  to  prevent  release  and/or  firing  of 
weapons  into  the  UMS  structure  or  other  weapons. 

DSP-6*:  The  UMS  shall  be  designed  to  prevent  uncommanded  fire  and/or 
release  of  weapons  or  propagation  and/or  radiation  of 
hazardous  energy. 

DSP-7*:  The  UMS  shall  be  designed  to  safely  initialize  in  the  intended 
state,  safely  and  verifiably  change  modes  and  states,  and 
prevent  hazardous  system  mode  combinations  or  transitions. 


33 


Design  Safety  Precepts 

(Cont’d) 


DSP-8*:  The  UMS  shall  be  designed  to  provide  for  an 

authorized  entity(s)  to  abort  operations  and  return  the 
system  to  a  safe  state,  if  possible. 

DSP-9*:  Safety  critical  software  for  the  UMS  design  shall  only 
include  required  and  intended  functionality. 

DSP-10*:  The  UMS  shall  be  designed  to  minimize  single¬ 
point,  common  mode  or  common  cause  failures 
that  result  in  high  and/or  serious  risks. 

DSP-11*:  The  UMS  shall  be  designed  to  minimize  the  use 
of  hazardous  materials. 

DSP-12*:  The  UMS  shall  be  designed  to  minimize 
exposure  of  personnel,  ordnance,  and 
equipment  to  hazards  generated  by  the  UMS 
equipment. 

DSP-13*:  The  UMS  shall  be  designed  to  identify  to  the 
authorized  entity(ies)  the  weapon  being 
released  or  fired,  but  prior  to  weapon  release  or  fire. 


Design  Safety  Precepts 

(Cont’d) 


DSP-14*:  In  the  event  of  unexpected  loss  or  corruption  of 
command  link,  the  UMS  shall  transition  to  a  pre¬ 
determined  and  expected  state  and  mode. 

DSP-15*:  The  firing  of  weapons  systems  shall  require  a 

minimum  of  two  independent  and  unique  validated 
messages  in  the  proper  sequence  from  the  authorized 
entity(ies),  each  of  which  shall  be  generated  as  a 
consequence  of  separate  authorized  entity  action. 
Both  messages  should  not  originate  within  the  UMS 
launching  platform. 

DSP-16:  The  UMS  shall  be  designed  to  provide  contingencies 
in  the  event  of  safety  critical  failures  or  emergencies 
involving  the  UMS. 

DSP-17:  The  UMS  shall  be  designed  to  ensure  safe  recovery  of 
the  UMS. 

DSP-18*:  The  UMS  shall  ensure  compatibility  with  the  test  range 
environment  to  provide  safety  during  test  and 
evaluation. 

DSP-19*  The  UMS  shall  be  designed  to  safely  operate  within 
combined  and  joint  operational  environments. 
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Precept  Clarification  Table 


Precept  Number:  Statement  of  the  precept  in  the  form  of  a 
requirement  or  general  guidance. 

Scope:  Answers  the  question  of  “What?”  the  precept  is  for;  often 
can  be  answered  by  “This  precept  addresses....” 

Rationale:  Answers  the  question  of  “Why?”  the  precept  is  required. 
This  provides  addition  clarification  of  the  intent  of  the  precept. 

Example:  Provide  as  many  clarifying  explicit/real-world  examples  to 
demonstrate  the  issues  and  specific  hazards  the  precept  addresses. 

Detailed  Considerations:  Answers  the  question  of  “How?”  by 
providing  details  to  assist  with  implementation  of  the  precept.  These 
are  specific  statements  written  in  the  form  of  a  requirement  or 
guideline  which  capture  lessons  learned  and  experience  from  other 
programs.  Some  of  these  considerations  can  be  tailored  for  specific 
programs  and  incorporated  into  system  specifications  as  safety 
requirements. 


DSP-14  Loss  of  Command  Link 


DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of 
command  link,  the  UMS  shall  transition  to  a  pre-determined  and 
expected  state  and  mode. 

Scope:  This  precept  addresses  the  overall  UMS  design  architecture  and 
states  and  mode  management  in  the  event  of  unexpected  loss 
or  corruption  of  the  command,  control,  and  communications  link  (i.e. 
loss  of  data  link,  loss  of  command  and  control).  The  objective  is  for  the 
UMS  to  be  in  the  anticipated/expected  state  when  recovery  occurs.  It  is 
not  the  intended  communication  loss  as  in  the  case  of  underwater 
vessels  or  other  fully  autonomous  UMS.  The  system  should  have  the 
capability  of  storing  a  set  of  actions  to  take,  or  states  to  transition  to, 
when  the  command  link  is  lost.  Predetermined  means  we  have  them  in 
the  plan.  Expected  means  we  intend  that  portion  of  the  plan  to  go  into 
effect  for  this  condition.  It  applies  to  both  the  test  and  perational 
environments.  This  precept  is  related  to  DSP-3  and  DSP-16. 

Rationale:  The  intent  of  this  precept  is  to  assure  that,  by  design;  the 
controlling  entity  can  anticipate  the  status,  mode  and  state  of  the 
UMS,  and  any  on-board  weapons  during  a  loss  of  link  period,  corruption 
of  link,  and  the  subsequent  recovery  of  link.  Determination  of  pre¬ 
determined  and  expected  status  should  be  based  on  analysis  of  such 
things  as  CONOPS,  mission  profile,  and  threat  hazard  assessments. 


DSP-14  Loss  of  Command  Link 

(cont’d) 


DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of 
command  link,  the  UMS  shall  transition  to  a  pre-determined 
and  expected  state  and  mode. 

Examples: 

1.  A  UAV  would  continue  to  fly  out  of  range  upon  loss  of 
command  link  if  no  contingency  provisions  are  designed  into 
the  system. 

2.  A  UAV  has  been  directed  upon  loss  of  link  to  return  to  base. 
It  currently  has  mission  parameters  loaded,  weapons  have  been 
energized,  and  commanded  to  fire  when  communications  link 
has  been  lost.  The  UAV  responds  to  its  mission  parameters 
and  is  returning  to  base  when  it  re-establishes 
communications. ...what  state  are  the  weapons  in?  Will  it  now 
execute  its  command  to  fire?  If  communications  are  lost  and 
re-established,  the  UAV  and  weapons  should  default  to  an 
expected  state. 


DSP-14  Loss  of  Command  Link 

(cont’d) 


DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of  command 
link,  the  UMS  shall  transition  to  a  pre-determined  and  expected 
state  and  mode. 


Detailed  Considerations: 

•  The  design  should  define  state  and  mode  transitions,  including 
a  desired  and/or  predictable  course  of  action  (such  as  move 
physically  to  a  safe  zone  or  crash  in  a  safe  zone),  in  the  event 
of  loss  of  link  or  intermittent  command  and  control.  The  criteria 
for  pre-determined  and  expected  states  and  modes,  and  the 
courses  of  action  include: 

-  the  UMS  CONOPS  and  application; 

-  the  level  of  autonomy  and  level  of  control; 

-  the  operating  environment  (i.e.  training,  test,  underwater, 
airborne,  etc.); 

-  the  adequacy  of  communication  link. 


DSP-14  Loss  of  Command  Link 

_ (cont’d) _ 

DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of  command 
link,  the  UMS  shall  transition  to  a  pre-determined  and  expected 
state  and  mode. 

Detailed  Considerations:  (cont’d) 

The  UMS  design  should  consider  retention  of  pertinent  mission 
information  (such  as  last  known  state  and  configuration,  etc.) 
for  the  UMS  and  the  controlling  entity(ies)  to  recover  from  loss 
of  the  communications  link. 

•  The  UMS  design  must  consider  limiting  the  duration  for  which 
undelivered  messages  are  considered  valid. 

•  The  UMS  design  must  consider  undelivered  messages  that  can 
exist  within  the  communication  system. 

•  The  UMS  should  ensure  command  messages  are  prioritized  and 
processed  in  the  correct  sequence  and  in  the  intended  state  and 
mode. 

•  Reference  NATO  STANAG  4404  Section  7.4  and  8.3.  DoD  8500.1 
Section  4.1;  and  DoD  5000.1  Section  El. 1.9. 


DSP-14  Loss  of  Command  Link 
_ (cont’d) _ 

DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of  command 
link,  the  UMS  shall  transition  to  a  pre-determined  and  expected 
state  and  mode. 

Existing  Policy: 


Service 

Document 

Section 

Comment 

Navy 

NAVSEA  SWO20-AH-SAF-1 0 

Section  14.8.3 

Text  partially 
references  precept. 

Need  your  help  in  identifying  any  other  existing 

policy  documents 

Protector  Unmanned  Surface  Vehicle 


Summary 

•S  Held  three  workshops  (March,  May,  June  2006) 

S  Government/industry/academia  teams  developed  draft 
safety  precepts,  rationale  &  design  guidance 

>  All  Services  and  numerous  UMS  program  office  reps 
participating 

■S  Briefed 

>  International  Systems  Safety  Conference  (2005,  2006 
and  2007) 

>  AUVSI  (August  2006) 

>  NDIA  Systems  Engineering  (October  2006  and  2007) 

•S  Comments  Requested 

>  NOSSA  Website 

(http://www.ih.navy.mil/unmannedsystems) 


xM-PIa 


Summary  (cont’d) 


USD  (AT&L)  Memorandum  of  17  July  2007 

Forwarded  the  Guide  to  the  Service  Secretaries  and  other  major  DoD 
components  as  an  enclosure  to  a  memo  strongly  endorsing  the  use  of 
the  Guide  for  all  UMS  acquisitions. 

S  The  Undersecretary  directed  that  the  UMS  Safety  precepts  in  the  Guide 
be  a  special  interest  item  for  ACAT  ID  Program  Support  Reviews. 


S  The  Guide  has  been  posted  on  the  OSD  ATP-TF  Website  at 
http://www.acq.osd.mil/atptf/ 

S  Next  steps: 

^  Convert  the  Guide  to  a  MIL-HDBK 

•  Handbook  is  for  guidance 

•  Service  ownership 

•  Facilitate  periodic  updates 

•  Formatting  completed  September  2007 

•  Final  Handbook  completion  3rd  Qtr  2008 

^  Update  Policy  and  Service  Directives  to  address  UMS  Precepts,  where 
appropriate.  (Remember,  12  Safety  Precepts  not  addressed  at  all  in  policy.) 
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Safety  of  Unmanned  Systems 


Sponsored  by 


CMMI  for  Services: 

Re-introducing  the  CMMI  for 
Services  Constellation 


NDIA  Systems  Engineering  Conference 

October  22-25,  2007 

Craig  R.  Hollenbach 
Northrop  Grumman  Corporation 

Brandon  Buteau 
Northrop  Grumman  Corporation 

Drew  Allison 

Systems  and  Software  Consortium  Inc. 

Frank  Niessink 
DNV-CIBIT 


CMMI 


Agenda 


CMMI 


•  CMMI-SVC  News 

•  Overview  of  the  draft  CMMI  for  Services  (CMMI-SVC) 

•  What  is  the  CMMI? 

•  Why  is  the  CMMI-SVC  needed? 

•  How  are  services  different? 

•  What  is  the  basis  for  the  CMMI-SVC  model? 

•  What  is  the  scope  and  content  of  the  CMMI-SVC? 

•  Feedback  to  date 

•  What  was  the  result  of  the  expert  review? 

•  What  was  the  experience  of  the  pilot  projects? 

•  Next  Steps 

•  What  is  the  schedule? 

How  can  I  participate? 
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CMMI  Steering  Group  to 
Address  CMMI  for  Services 


CMMI 


•  There  was  a  serious  concern  that  concurrent 
development  of  the  CMMI-ACQ  and  CMMI-SVC 
models  would  stress  the  SEI  resources  needed  to 
deliver  the  CMMI-ACQ  model  on  time.  Now  that 
CMMI-ACQ  is  almost  released,  the  SEI  resources 
are  available  to  go  forward  with  the  CMMI-SVC. 

•  The  CMMI-SVC  team  will  address  past  Steering 
Group  concerns  at  their  Nov  meeting  and  present 
a  plan  to  complete  development. 


What  is  a  Capability  Maturity 
Model  (CMM)? 


CMMI 


•  A  conceptual  framework  for  structuring,  understanding,  and 
evaluating  the  capability  and  maturity  of  an  organization’s 
processes 

•  more  than  a  laundry  list  of  best  practices 

•  more  than  a  collection  of  benchmarks  and  metrics 

•  A  tool  that  enables  meaningful,  in-depth  organizational 
assessment 

•  internally 

•  externally 

•  A  map  that  guides  practical  process  improvement  and 
institutionalizes  it 

How  to  you  get  from  here  to  there  and  stay  there ? 


What  is  the  CMMI? 


CMMI 


The  CMM  Integration  (CMMI)  of  multiple  CMMs  into  a 
single  unified  framework 


EIA  Interim  Standard  731, 
System  Engineering 
Capability  Model  (SECM) 


Capability  Maturity 
Model  for  Software  V2, 
draft  C  (SW-CMM  V2C) 


Integrated  Product 
Development 
Capability  Maturity 
Model,  draft  V0.98 
(IPD-CMM) 


Software  Acquisition 
Capability  Maturity  Model, 
version  1.01  (SA-CMM) 


■  CMMI  1 

Product  Suite  | 

Three  complementary 
constellations 


CMMI 


CMMI-DEV 

CMMI-SVC 

provides  guidance 

provides  guidance  for 

for  measuring, 

those  providing 

monitoring,  and 

services  within 

managing 

organizations  and  to 

development 

external  customers 

processes 

Core 
Model 
Foundation 


CMMI-ACQ 

provides  guidance 
to  enable 
informed  and 
decisive 
acquisition 
leadership 


Courtesy  of  the  SEI 


5 


Why  is  CMMI  for  Services 
(CMMI-SVC)  needed? 


CMMI 


y 


•  Customer  discontent 

•  Service  society 

•  Legislation 

•  Government  and  industry 
trends 
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How  are  services  different? 


CMMI 


•  Services  form  a  distinctive  category  of  products 

•  A  service  is  an  intangible,  non-storable  product 

•  What  makes  a  service  intangible  or  non-storable? 

•  Customer  desires  a  situation  or  state  (e.g.,  to  have  high  network 
availability)  rather  than  a  tangible  artifact 

•  Provider  delivers  value  without  allowing  the  customer  independent, 
unrestricted  means  to  generating/employing  that  value  (e.g.,  leasing 
vehicles) 

•  Product  delivery  requires  continuing  application  of  labor  (e.g.,  operation 
of  a  facility) 

•  Services  imply  customer/provider  relationships  governed  by 
service  agreements 

Service  and  non-service  products  may  be  delivered  as  part  of  a  single 

agreement  (e.g.,  training  that  includes  hardcopy  materials) 

•  Services  are  often  delivered  via  the  operation  of  a  service  system 


Service  system 


CMMI 


•  A  necessary  concept  for  understanding  the  effective 
delivery  of  services 

•  An  integrated  and  interdependent  combination  of 
processes,  resources,  and  people  that  satisfies  service 
requirements. 

•  Portions  are  not  delivered  to  the  customer  or  end-user  as 
part  of  service  delivery 

•  Portions  may  remain  owned  by  the  customer  or  end-user 
before  service  delivery  begins  and  after  service  delivery 
ends. 

•  Encompasses  everything  required  for  service  delivery, 
including  work  products,  processes,  infrastructure, 
consumables,  and  customer  resources. 
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What  is  the  scope  of 
CMMI-SVC? 


CMMI 


•  Covers  practices  required  to  manage,  establish,  and  deliver 
services,  in  four  process  area  categories 

•  Project  (service)  management 

•  Process  management 

•  Service  support 

•  Service  establishment  and  delivery 

•  Intended  to  match  the  scope  of  the  definition  of  services 

•  Broad  applicability  to  a  range  of  service  domains 

•  Information  technology,  engineering,  defense,  transportation, 
finance,  health  care 

•  Staff  augmentation  services  need  careful  consideration 

•  How  do  you  evaluate  process  improvement  for  processes  over 
which  you  have  no  control? 


CMMI-SVC  Process  Areas 


CMMI 


Process  Management 

Organizational  Innovation  and 
Deployment  (OID) 

Organizational  Process  Definition  (OPD) 
Organizational  Process  Focus  (OPF) 

Organizational  Process  Performance 
(OPP) 

Organizational  Service  Management 
(OSM) 

Organizational  Training  (OT) 

Service  Support 

Causal  Analysis  and  Resolution  (CAR) 
Configuration  Management  (CM) 
Decision  Analysis  and  Resolution  (DAR) 
Measurement  and  Analysis  (MA) 

Problem  Management  (PRM) 

Process  and  Product  Quality  Assurance 
(PPQA) 


Service  Establishment  and  Delivery 

•  Incident  and  Request  Management 
(IRM) 

•  Service  Delivery  (SD) 

•  Service  System  Development  (SSD) 

•  Service  Transition  (ST) 

Project  Management 

•  Capacity  and  A  vailability  Management 
(CAM) 

•  Integrated  Project  Management  (IPM) 

•  Project  Monitoring  and  Control  (PMC) 

•  Project  Planning  (PP) 

•  Requirements  Management  (REQM) 

•  Risk  Management  (RSKM) 

•  Quantitative  Project  Management  (QPM) 

•  Service  Continuity  (SCON) 

•  Supplier  Agreement  Management  (SAM) 
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CMMI 

Services-specific  PAs 


Process  Area 

Maturity  Level 

Specific  Goals/ 
Practices 

Capability  and  Availability  Management  (CAM) 

3 

2/6 

Incident  and  Request  Management  (IRM) 

2 

2/6 

Organizational  Service  Management  (OSM)* 

3 

2/7 

Problem  Management  (PRM) 

3 

2/7 

Service  Continuity  (SCON)* 

3 

3/10 

Service  Delivery  (SD) 

3 

2/7 

Service  System  Development  (SSD)  * 

3 

3/12 

Service  Transition  (ST) 

3 

3/12 

*  optional  process  areas  (independent  named  additions) 
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CMMI-SVC  Level  2  PAs 


CMMK 


•  Incident  and  Request  Management 

•  To  ensure  the  timely  resolution  of  requests  for  service 
and  incidents  that  occur  during  service  delivery 

•  Requirements  Management 

•  Extended  from  the  Core  Model  Foundation  with  an 
additional  goal 

•  To  include  the  establishment  and  maintenance  of  written 
agreements  between  service  providers  and  customers 
on  service  requirements  and  service  levels. 

•  Six  other  level  2  PAs  from  the  CMF 
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CMMI-SVC  Level  3  PAs 


CMMK 


•  Capacity  and  Availability  Management 

•  To  plan  and  monitor  the  effective  provision  of  resources 
to  support  service  requirements 

•  Problem  Management 

•  To  prevent  incidents  from  recurring  by  identifying  and 
addressing  underlying  causes  of  incidents 

•  Service  Delivery 

•  To  deliver  services  in  accordance  with  service 
agreements 

•  Service  Transition 

•  To  deploy  new  or  significantly  changed  service  systems 
while  managing  their  effect  on  ongoing  service  delivery 
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Optional  PAs  for  CMMI-SVC 
Level  3 


CMMI 


•  Organizational  Service  Management 

•  To  establish  and  maintain  standard  services  that  ensure 
the  satisfaction  of  the  organization's  customer  base 

•  Service  Continuity  Management 

•  To  establish  and  maintain  contingency  plans  for 
continuity  of  agreed  services  during  and  following  any 
significant  disruption  of  normal  operations 

•  Service  System  Development 

•  To  analyze,  design,  develop,  integrate,  and  test  service 
systems  to  satisfy  existing  or  anticipated  service 
agreements 
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What  was  the  result  of  the 
expert  review? 


CMMI 


•  An  expert  review  was  held  Jan  23  -  Mar  23,  2007 

•  500+  reviewers,  representing: 

•  50  companies, 

•  14  DoD  organizations, 

•  4  academic  institutions,  and 

•  7  professional,  governmental,  or  research  centers 

•  Reviewers  included  SEI  transition  partners 

•  Response  showed  strong  interest  in  CMMI-SVC 

•  900+  change  requests  compares  favorably  to  those 
received  for  CMMI-DEV 

•  50  survey  responses  to  architectural  questions 
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What  was  the  result  of  the 
expert  review?  (more) 


CMMI 


•  Reviews  commented  mostly  on  CMM-SVC  architecture  &  Common 
Model  Foundation  material 

•  CRs  were  distributed  equally  among  categories  related  to  SVC  PAs 

•  CMMI-SVC  team  has  analyzed  all  architectural  CRs;  most  have  a 
proposed  resolution 

•  CRs  showed  excellent  depth  of  insight  and  rich  informative  content 
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CMMI 

Sample  Survey  Responses 

•  The  service  practices  that  are  covered  in  CMMI-SVC  will  enable  service  organizations  to  provide  more 
effective  support  to  their  customers. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

78.9% 

8.8% 

12.3% 

•  The  material  in  CMMI-SVC  yields  a  useful  adaptation  of  CMMI  best  practices  as  they  relate  to  service 
deployment. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

66.7% 

14.0% 

15.8% 

•  CMMI-SVC  does  not  impose  constraints  (derived  from  the  needs  of  a  specific  service  or  market 

segment)  that  would  limit  or  prevent  other  organizations  from  adapting  the  model  to  their  own  specific 
needs. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

55.6% 

29.6% 

27.8% 

•  The  CMMI-SVC  is  easy  to  understand  and  apply  to  a  service  organization. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

42.8% 

27.8% 

29.6% 
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What  was  the  experience  of 
the  pilot  projects? 


CMMI 


•  Planned  pilots  were  postponed 

•  CMMI-SVC  participating  companies  piloted  the  model  internally 

•  Characteristics  of  the  piloted  organizations: 

•  Most  had  implemented  CMMI-DEV 

Some  had  separate  ITIL  and  ISO  20000  initiatives 
Most  are  moving  towards  integration  under  CMMI  umbrella 

•  The  pilots  represented  the  following  service  domains: 


Company 

Service  Domains 

SSCI 

IT  Application  Operations  &  Support 

DNV-CIBIT 

Banking 

Northrop  Grumman 

Logistics,  HR,  IT,  Applications  O&M 

18 


What  did  the  pilots  see  as 
benefits? 


CMMI 


•  Improved  quality  of  services 

•  Encouraged  a  disciplined  culture  for  service  management 

Better  management  visibility  into  services 
Fewer  surprises 

•  Fosters  process  improvement 

•  Less  Interpretation  issues  (&  appraisal  expense)  than  with  CMMI-DEV 

•  Applying  a  CMMI  process  to  the  services  brought  credibility  and  buy-in 
from  stakeholders 

•  Increased  sharing  between  development  and  services  communities 

•  Common  processes 

•  Standard  terminology 

Integrated  process  improvement  standards  and  models 

•  Encouraged  end-to-end  lifecycle  process  approach  helping  to  identify 
service  requirements,  ease  deployment  issues,  reduce  stove-piped 
groups,  and  improve  efficiencies  of  support-related  groups  (IT 

Appl  cations) 
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What  did  the  pilots  see  as 
challenges? 


CMMI 


•  Obtaining  funding  in  environments  that  are  primarily  LOE-based 

•  Differences  in  terminology  between  development  and  services 

Terms  like  “Project”  (funding  period),  “Product”  (service),  “Work  Product”, 
“Product  Component”,  “Requirement” 

Interpreting  CMMI’s  “project”  term  for  services 

•  No  standard  life-cycle  definition  for  services 

•  Instilling  project  management  culture  in  services 

Weak  in  using  requirements  for  planning  and  negotiating  resources  and 
activities 

•  Ownership  of  service  system  components  not  as  clear 

•  Release  management  and  deployment  to  non-standardized,  constantly 
changing  environments 

•  Finding  CMMI-knowledgeable  individuals  who  also  know  services 

•  Integrating  process  groups  and  assets 

•  Services  where  customer  and  provider  share  resources  and  processes 

•  Staff  augmentation 
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Issues  to  Address 


CMMI 


•  What  is  the  business  case  for  the  CMMI-SVC? 

•  What  distinguishes  CMMI-SVC  from  CMMI-DEV  (vl  .2)  and 
other  models? 

•  What  are  the  characteristics  of  service  providers  and  how 
are  they  represented  in  the  CMMI-SVC? 

•  Can  the  broad  spectrum  of  services  be  governed  by  a 
single  model? 

•  How  will  the  Services  Sector  be  engaged? 

•  What  are  the  impacts  to  small  businesses? 

•  How  will  CMMI-SVC  be  used  with  other  CMMI  products? 
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What  is  the  schedule? 


CMMI 


•  CMMI-SVC  team  will  meet  to  review  additional 
requirements  and  re-plan  remaining  work  (early  Nov) 

•  Detailed  schedule  is  pending 

•  A  preliminary  estimate  for  release  of  CMMI-SVC,  vl  .2  is 
4th  quarter  2008 


ID 

Task  Name 

4 

CMMI-SVC,  v0.5 

5 

CMMI-SVC,  v0.5  review 

6 

CMMI-SVC,  vl  .2 

2005 


Qtrl  Qtr  2  Qtr  3  Qtr  4 


2006 


Qtrl  Qtr  2  Qtr  3  Qtr  4 


2007 


Qtrl  !  Qtr  2  Qtr  3  Qtr  4 


2008 


Qtrl  Qtr  2  Qtr  3  Qtr  4 
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How  can  I  participate? 


CMMI 


•  Get  more  information  about  CMMI-SVC 

•  CMMI  web  page  -  http://www.sei.cmu.edu/cmmi/ 

•  CMMI  for  Services  Public  Workspace 

(http://bscw.sei.cmu.edU/bscw/bscw.cai/0/424939)  contains: 

•  Draft  CMMI-SVC  model,  v0.5 

•  Information  on  joining  CMMI-SVC  information  email  list 

•  Review  draft  CMMI-SVC  release 

•  If  already  experienced  in  CMMI,  consider  piloting  the  model 

•  Other  opportunities  may  exist  as  a  result  of  the  CMMI-SVC 
re-planning  effort;  watch  CMMI-SVC  public  workspace  for 
updates 
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Backup 


CMMI 
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CMMI 
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Who  is  working  on 
CMMI-SVC? 


•  Development  Team 

Craig  Hollenbach  (Northrop  Grumman) 
Roy  Porter  (Northrop  Grumman) 
Brandon  Buteau  (Northrop  Grumman) 
Lynn  Penn  (Lockheed  Martin) 

•  Frank  Niessink  (DNV/CIBIT) 

•  Jerry  Simpson  (SAIC) 

•  Drew  Allison  (SSCI) 

Eileen  Forrester  (SEI) 

•  Barbara  Tyson  (SEI) 

•  Eileen  Clark  (SRA) 

•  Other  contributors 

Jeff  Zeidler  (Boeing) 

•  Rich  Raphael  (Mitre) 

Joanne  O’Leary  (SEI) 
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CMMI 

General  Survey  Questions 


1.  The  service  practices  that  are  covered  in  CMMI-SVC  will  enable  service  organizations  to 
provide  more  effective  support  to  their  customers. 

2.  The  material  in  CMMI-SVC  yields  a  useful  adaptation  of  CMMI  best  practices  as  they 
relate  to  service  deployment. 

3.  The  CMMI-SVC  appropriately  uses  the  CMMI  framework. 

4.  CMMI-SVC  includes  process  areas  that  must  be  satisfied  for  process  improvement  and 
institutionalization. 

5.  CMMI-SVC  does  not  impose  constraints  (derived  from  the  needs  of  a  specific  service  or 
market  segment)  that  would  limit  or  prevent  other  organizations  from  adapting  the  model 
to  their  own  specific  needs. 

6.  The  CMMI-SVC  is  easy  to  understand  and  apply  to  a  service  organization. 

7.  The  process  areas  in  CMMI-SVC  cover  all  significant  service-specific  requirements  and 
effectively  reflect  activities  that  a  service  organization  should  be  accomplishing. 

8.  Additions  and  amplifications  that  exist  in  other  models  and  are  also  used  within  the  CMMI- 
SVC  constellation  are  appropriate. 

9.  Notes  and  examples  in  CMMI-SVC  clearly  apply  to  service  organizations  and  meet  their 
specific  needs. 

10.  References  in  PAs  to  related  process  areas  are  clear  and  consistently  applied. 
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CMMI 

Results  to  General  Survey 


Survey  Responses 


□  Agree  or  Strongly  Agree 

□  Neutral 

□  Disagree  or  Strongly 
Disagree 
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CJWMI 

Process  Area  Questions 


a.  Problem  management  practices  that  are  common  within  the  service  industry  are  appropriately 
addressed  in  the  process  area  Problem  Management  and  are  distinguished  from  the  practices  in  the 
Causal  Analysis  and  Resolution  process  area. 

b.  The  Project  Management  category  is  the  most  appropriate  classification  for  the  Service  Continuity 
Management  and  Capacity  and  Availability  Management  process  areas. 

c.  The  Process  Management  category  is  the  most  appropriate  classification  for  the  Organizational 
Service  Management  process  area 

d.  The  practices  within  the  Service  Continuity  process  area  should  build  upon  the  practices  within  the 
Risk  Management  process  area  similar  to  the  manner  in  which  the  Integrated  Project  Management 
process  area  builds  upon  maturity  level  2  project  management  practices. 

e.  The  Service  System  Development  process  area  must  be  required  for  an  organization  to  be  a  mature 
service  organization. 

f.  The  specific  practices  in  the  Service  System  Development  process  areas  are  presented  with  the 
appropriate  rigor  and  detail  for  a  mature  service  organization. 

g.  The  Project  Monitoring  and  Control  process  area  adequately  addresses  service  level  management. 

h.  Material  about  the  collection  of  customer  satisfaction  information  is  adequately  covered  as  a  specific 
practice  in  Organizational  Service  Management  (an  optional  process  area)  and  as  informative  material 
in  the  Service  Delivery  process  area. 

i.  Maintenance  found  in  the  Service  Delivery  process  area  is  adequately  differentiated  from  product 
maintenance  covered  by  CMMI-DEV. 

j.  The  IPPD  addition  is  as  appropriate  or  as  applicable  for  CMMI-SVC  as  it  is  for  CMMI-DEV  and  should 
be  added. 

k.  The  Supplier  Agreement  Management  process  area  is  appropriate  both  for  organizations  with  tangible 
products  and  service  organizations  with  supplier  agreements  solely  for  services. 

l.  The  Supplier  Agreement  Management  process  area  should  be  required  to  reach  maturity  level  2  for 
service  organizations  with  supplier  agreements  solely  for  services  (as  it  is  for  organizations  with 
suppliers  of  tangible  products). 
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Process  Area  Survey  Questions 


CMMI 


Process  Area  Survey  Questions 


■  Agree  or  Strongly  Agree 
□  Neutral 

H  Disagree  or  Strongly 
Disagree 
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What  is  the  relationship 
between  CMMI-SVC  and  ITIL? 


CMMI 


•  CMMI-SVC  complements  ITIL 

•  Summarizes  ITIL  best  practices  into  a  small  set  of 
specific  practices. 

•  Reuses  about  80%  of  the  current  CMMI  model,  allowing 
users  to  leverage  their  investments  in  development- 
based  process  training,  improvements,  and  infrastructure 
to  serv  ce-based  offerings. 

•  Provides  an  industry-accepted  maturity  model,  helping 
organizations  to  plan  and  track  their  incremental 
progress  toward  high  maturity. 

•  Uses  the  same  SCAMPI  appraisal  method  that  is  used 
with  the  current  CMMI  model,  allowing  organizations  to 
leverage  appraisal  expertise,  preparation  methods,  and 
selected  artifacts. 
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Who  uses  CMMs? 


Commercial/In-house 


67.6% 


Contractor  for 
Military /Government 


28.8% 


Military/Government 

Agency 


0  200  400  600  800  1000 

Number  of  Organizations 


CMMI 


1200 


Courtesy  of  the  SEI 
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Why  do  CMMs  really  matter? 


Improvements 

Median 

Data 

Count 

Low 

High 

Cost 

34% 

29 

3% 

87% 

Schedule 

50% 

22 

2% 

95% 

Productivity 

61% 

20 

11% 

329% 

Quality 

48% 

34 

2% 

1 32% 

Customer 

Satisfaction 

14% 

7 

-4% 

55% 

ROI 

4.0  :  1 

22 

1.7  :  1 

27.7  :  1 

CM  Ml 


•  N  =  30,  as  of  August  2006 

•  Organizations  with  results  expressed  as  change  over  time 


Courtesy  of  the  SEI 
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Anatomy  of  an  Award  Winning  Safety  Program: 

A  Case  Study  of  the 

SSGN  OHIO  Class  Conversion  Safety  Program 


Mike  Parulis  (for)  Thomas  Cook 

& 


Ricky  Milnarik  general  dynamics 

Electric  Boat 


i 


Snip  Characteristics 


No.  of  SOF  Personnel 
No.  of  VLS  Missiles 
DDS/ASDS  Capability 


66  to  102 


Up  to  154 


20+  knots 


K 


154  Strike  Missile 
Tomahawk/TACTOM 


Dual  Advanced  SEAL  Delivery  System  (ASDS)  and 
Drv  Deck  Shelter  (DDS)  Caaabilitv 


rJ  r  J 

DD 


GENERAL.  DYNAMICS 

Electric  Boat 


•  154  TOMAHAWK  Missiles 

•  66  Special  Operations  Forces  (SOF)  for  more  than  60  Days 

•  2  Dry  Deck  Shelter  /  Advanced  SEAL  Delivery  System 
•  8  Modular  SOF  Storage  Canisters 
•  Battle  Management  Center: 

Joint  Connectivity  and  Organic  Command  &  Control  Capability 

•  Communications  suite  has  double  the  antennas  of  an  SSN 

•  SOF  Habitability  &  Training  Facilities 
•  SEASUB  -  Lock-ln/Lock-Out,  Ordnance  Package 


SSGN  Conversion 
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A  NAVSEA  (Program  Office) 

Perspective 


Introduction 


NAVSEA  assembled  multi-disciplinary  teams  that 
developed  and  implemented  ESOH  management 
programs  whose  goals  were  to  incorporate  life 
cycle  ESOH  compliance  into  design  and 
construction 
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SSGN  Conversion  ESOH 


*  ADHOC  -  Participants  as  necessary  (i.e.,  NOSSA  /  WSESRB,  Legacy,  Shipboard  Shock,  Packaging,  Handling  and  Transportation) 
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Electric  Boat 


SSGN  Conversion  ESOH 
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Electric  Boat 


STEMS  COMMAND 


SSGN  Conversion  ESOH 


•  MIL-STD-882D  requirement 

-  Allows  flexibility 

•  Multiple  Government  Agencies 

•  Prime  Shipbuilder  Contractor 

•  Several  other  Contractors 

-  Each  organization  has  own  safety  plan 

•  All  are  in  accordance  with  program  office  plan 

•  Each  organization  plan  is  tailored  for  specific 
“way  of  doing  business” 
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SSGN  Conversion  ESOH 


•  Integrated  Product  Team  (IPT)  Process 

-  Not  exactly  an  IPT  process,  but  encompasses  the  open 
communication 

•  Open  expression  of  ideas  (safety  and  process) 

•  Members  encouraged  to  voice  concerns 

•  Seeks  consensus  on  programmatic  issues  and  processes 

-  Buy-In  by  the  program  office,  Principal  for  Safety, and  the 
individual  (contractor  or  government)  identifying  the 
Hazard 

•  Identification  -  at  least  three  signatures 

•  Acceptance  -  at  least  four  signatures 
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SSGN  Conversion  ESOH 


•  Performance  based  Specification  instead  of 
detailed  requirements 

-  As  Principal  for  Safety 

•  Insist  on  end-results 

-  Hazards/Impacts  identified  openly  (i.e.,  don’t  suppress) 

-  Hazards/Impacts  mitigated  in  a  consistent  manner 

»  Each  organization  follows  the  same  MIL-STD-882 
logic  for  severity  and  probability  (Initial  &  Final) 

•  Do  not  dictate  manner  to  reach  end-results 

-  Analyses  conducted  by  each  organization  as  per  their 
processes 
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NAVSEA  Summary 


•  Program  Support 

-  Continuous  funding 

-  Adequate  safety  manning 

•  Safety  IPT  Independence 

•  Contributing  organizations  staffed  by  experience 
safety  engineers 

-  Strategic  Systems  Programs  (e.g.,  TRIDENT) 

-  Naval  Undersea  Warfare  Center 

-  Naval  Air  Systems  Command 

-  Electric  Boat  Corporation 

-  Numerous  Sub-Contractors 
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A  Contractors  Perspective 


:icky  Milnarik 
ISsmineerfinci  Specialist 
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Electric  Boat 
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Electric  Boat  Corporation 


Electric  Boat  has  been  building  submarines  for  the 
U.  S.  Navy  for  over  100  years. 


In  1900  Electric  Boat 
delivered  the 
U.  S.  Navy’s  first 
submarine,  the 
USS  Holland. 
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Electric  Boat 


Integrated  Product  and  Process 

Development  (IPPD) 

•  First  used  at  Electric  Boat  extensively  on  the 
VIRGINIA  Class  Submarine  program  in  1994 

•  Before  the  IPPD  process,  a  serial  approach  to 
submarine  design-to-construction  was  taken. 

•  The  dynamics  of  the  IPPD  process  is  made 
possible  through  the  use  of  the  Computer  Aided 
Three-Dimensional  Interactive  Application 
(CATIA)  software  design  tool  to  develop  electronic 
mockups. 
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Electric  Boat 


Integrated  Product  and  Process 

Development  (IPPD) 


•  Methodology  consists  of  activity-based  product 
management  and  concurrent  engineering  Design 
Build  Teams  (DBTs). 

•  Team  assignments  are  structured  in  accordance 
with  program  development  and  manufacturing 
needs. 
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Electric  Boat 


Design  /  Build  Teams 


A  typical  DBT  makeup  is  shown  below 


Operations 


Materials 


Design 
Build  Team 


Design 

System  Safety 
Engineering 

Environmen 

Purchasing 


Planning 

NAVSEi 


Quality  Assurance, 
Navy  Operators 
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Design  /  Build  Teams 


DBT  functional  managers  /  technical  leaders 
have  direct  management  and  control  of  their 

specific  functional  areas. 
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Before  SSGN  ! 


•  Prior  to  SSGN,  System  Safety  Engineering  and 
Environmental  Engineering  groups  at  Electric  Boat 
were  not  merged  into  a  single  Environmental, 
Safety,  and  Occupational  Health  (ESOH)  group. 

•  System  Safety  and  Environmental  Engineering 
were  separate  parallel  processes. 

•  System  Safety  and  Environmental  engineers  were 
in  separate  locations  7  miles  apart. 
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Before  SSGN  ! 


In  support  of  the  VIRGINIA  Class  IPPD  process: 

•  System  Safety  Engineering  conducted  traditional 
MIL-STD-882  hazard  analysis  reports  on  identified 
ship  systems. 

•  Environmental  Engineering  conducted  Design/Build 
Environmental  Analyses  (DBEA)  on  identified  ship 
systems. 


SEPARATE  PARALLEL  PROCESSES 
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Electric  Boat 


Before  SSGN  ! 


•  System  Safety  Engineering  identified  potential 
hazards. 

•  Environmental  Engineering  identified  potential 
environmental  impacts. 


SEPARATE  PARALLEL  PROCESSES 
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Before  SSGN  ! 


•  System  Safety  Engineering  tracked  hazards  in  an 
Hazard  Tracking  List  database. 

•  Environmental  Engineering  tracked  environmental 
impacts  in  a  DBEA  database. 


SEPARATE  PARALLEL  PROCESSES 
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SSGN  Conversion 


•  In  2001  the  SSGN  Conversion  Program 
provided  an  opportunity  to  eliminate  duplication 
and  integrate  System  Safety  and  Environmental 
Engineering  into  an  effective  ESOH  group 

•  Leveraging  off  the  lessons  learned  from  the 
VIRGINIA  (SSN  774)  Class  Submarine  Safety 
and  Environmental  programs,  the  SSGN 
Conversion  program  allowed  Electric  Boat  to 
implement  an  ESOH  program  per  DODI  5000.2 
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SSGN  Conversion  ESOH 


A  single  integrated  ESOH  Program  Plan  was 
developed.  Key  features  included: 

•  Making  ESOH  the  responsibility  of  the  DBT. 

•  Integrating  experienced  Safety  &  Environmental 
engineers  into  DBTs. 

•  Define  the  ESOH  hazard  analyses  for  all  Electric 
Boat  SSGN  Conversion  cognizant  systems. 

•  Establish  an  audit  trail  of  identified  ESOH  issues 
(safety  hazards/environmental  impacts). 
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SSGN  Conversion  ESOH 


Key  features  included  continued: 

•  The  system  safety  engineering  group  was  co-located 
with  the  environmental  engineering  group. 

•  An  ESOH  Program  Integrator  was  assigned  to  the 
program. 

•  A  single  report  format  that  would  satisfy  the  needs  of 
both  a  DBEA  and  MIL-STD-882  hazard  analysis  was 
developed. 


An  integrated  database  was  developed  to  track  both 
system  safety  hazards  and  environmental  impacts. 
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ESOH  Hazard  /  Impact  Database 


s  Microsoft  Access  -  [PRIMARY  :  Form] 


El  File  Edit  View  Insert  Format  Tools  Window  Help 


-3 


-=JjJ 


OHIO  CLASS  SSGN  CONVERSION  PROGRAM  ENVIRONMENT,  SAFETY, 
AND  OCCUPATIONAL  HEALTH  (ESOH)  HAZARD  /  IMPACT 
Version  1.0  TRACKING  DATABASE 


1000  Series 


2000  Series 


3000  Series 


4000  Series 


5000  Series 


ESOH  HAZARD  I  IMPACT  REPORTS 


(Prelim  =  Prior  to  PMS398  Concurrence)- 


(Open  +  Closed  =  All )  - 


ALL  SSGN  CONVERSION 
SYSTEMS 


ATTACK  WEAPON  SYSTEM 


ATTACK  WEAPON  SUPPORT 
SYSTEMS 


SOF  ORDNANCE  STOWAGE  AND 
HANDLING  SYSTEM 


SOF  DIVER  SUPPORT  SYSTEMS 


OTHER  SYSTEMS  AND 
EQUIPMENT 


ALL  ORDNANCE/WEAPON 
SYSTEM  RELATED 


ALL  SOF  SUPPORT  SYSTEMS 
RELATED 


SSGN  CONVERSION  SYSTEM 
SOFTWARE 


Pi  elim  I  Open 


Prelim 


Prelim 


Prelim 


Prelim 


Prelim 


Prelim 


Prelim 


Prelim 


SSGN  ENVIRONMENTAL 
MANAGER'S  REPORT 
SSGN  SAFETY  MANAGER'S 
REPORT 


PROCEDURE  ACTION  ITEM  LIST 
(PAIL) 


Prelim 


Pi  elim  I  Open 


Open 


Open 


Open 


Open 


Open 


Open 


Open 


Open 


Open 


Closed 


Closed 


Closed 


Closed 


Closed 


Closed 


Closed 


Closed 


Closed 


Closed 


Closed 


All 


All 


All 


All 


All 


All 


All 


All 


All 


All 


All 


ByDocumant  ■  ByESO 


Updated:  12  July  2002 


For  Analyst  /  Program 
Office  Use  Only 


DATA  INPUT  AND 
FORM  GENERATION 


r 


il 


View 
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ESOH  Hazard  /  Impact  Database 


The  hazard  /  impact  database  is  capable  of  accepting 
both  system  safety  hazards  and  environmental 
impacts  on  a  single  unique  Hazard/Impact 
Identification  Form. 
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Electric  Boat 


ESOH 

Hazard/Impact 

Identification 

Form 


1%  Microsoft  Access  [ESOH  HAZARD/IMPACT  IDENTIFICATION] 

HEIE3 

EH  File  Edit  View  Insert  Format  Records  Tools  Window  Help 

SSGN  PROGRAM  ENVIRONMENT.  SAFETY.  AND  OCCUPATIONAL 
HEALTH  (ESOH)  HAZARD  /  IMPACT  IDENTIFICATION  FORM 


System: 


Hazard  Impact  ID  llumber 


~ZJ 


Subsystem: 


Hazard  Impact  Title: 


Hazard  Impact  Description: 


Categories  of  Items  Relating  to  Hazard  /  Impact:  (Check  o 


e.)  |“  HARDWARE  1“  FIRMWARE  P  SOFTWARE 


Is  hazard/impact  related  to  Ordnance  Weapon  System?  |  No  ZJ 


Is  hazard/impact  related  to  Special  Operations  Forces?  |  No  ZJ 


Type  of  Hazard  /  Impact 

(Check  one  or  more .) 


Life  Cycle  Phase 

(Check  one  or  more.) 


I-  IIEPA 

|“  POLLUTION  PREVENTION 


P  ESOH  COMPLIANCE 
r  HAZ  MATERIAL  MGMT 


SAFETY  AND  HEALTH 
r  EXPLOSIVES  SAFETY 


r~  MANUFACTURE 
1“  INSTALLATION 


P  TEST  .EVALUATION 
r  OPERATION 


P  MAINTENANCE 
r  DISPOSAL 


Initial  Mishap  Severity  |  ZJ  Mishap  Probability  of  Occurrence  |  ' 3  Mishap  Risk  Assessment  Value  |  0  -  I 


Recommended  Mitigation  Action  or  Rationale  for  Accepting  without  Mitigation: 


Activity  Having  Lead  Responsibility  foi  Mitigation:  |”” 


Does  mitigation  eithei  affect  existing  procedures  or  require  new  procedures?  |  No  ZJ 


Pr°Fina|e<l  I  Mis,u1l>  Severity  rzi  Mishap  Probability  of  Occurrence  ZJ  Mishap  Risk  Assessment  Value  |  0  "I 


To  preview  the  IDENTIFICATION  FORM,  click  on  button.  To  print  the  form,  click  on 
the  print  icon  on  the  preview  screen . 

To  close  and  return  to  DATABASE  opening  screen,  click  on  button . 


PREVIEW  ID  FORM 


PROCEED  AS  FOLLOWS  AFTER  SYSTEM  DEVELOPMENT  MANAGER  SIGNS  IDENTIFICATION  FORM. 


Fill  out  AFTER  Identification  Form  has  been  signed  bv  SYSTEM  DEVELOPMENT  MANAGER. 

System  Development  Manager  Name  and  Activity  Date  Hazard-impact 

^Identifying  Hazard-impact: _  Identified: 

Click  on  the  " NEXT  ACTION  INPUT "  button  and  fill  out  the  information  on  the  'NEXT 
ACTION  TOWARD  HAZARD  IMPACT  CLOSURE  INPUT  FORM"  pertaining  to  this  NEXT 

hazard,  impact.  At  this  time,  denote  Status  of  Hazard  Impact  as  "Preliminary."  _ ii 


PROCEED  AS  FOLLOWS  AFTER  PROGRAM  OFFICE  (PMS398)  SIGNS  IDENTIFICATION  FORM. 


|  Fill  out  AFTER  Identification  Form  has  been  signed  by  PROGRAM  OFFICE. 

^Program  Office  Concurrence  of  Hazard impact  ID  (Name): _  Date  Concurred: 

I  If  mitigation  affects  existing  procedures  or  requires  new  procedures,  information 
I  concerning  the  procedures  must  be  entered  into  the  Procedure  Action  Item  List  (PAIL)  at 
this  time.  To  do  so,  click  on  "PAIL"  button  and  enter  the  required  data. 

|  Click  on  the  "NEXT  ACTION  INPUT"  button  and  update  the  information  on  the  "NEXT 
I  ACTION  TOWARD  HAZARD  IMPACT  CLOSURE  INPUT  FORM"  pertaining  to  this 
hazard,  impact.  At  this  time,  denote  Status  of  Hazard. Impact  as  "Open." 


Record:  H  |  <  1 1~ 


13  ►  I  H  !►*!  of  13 


[Unique  number  of  hazard/impact. 
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ESOH  Hazard  /  Impact  Database 


The  hazard  /  impact  database  can  generate  and  print: 

•  ESOH  Hazard/Impact  Identification  Forms 

•  ESOH  Hazard/Impact  Closure  Forms 

•  ESOH  Hazard/Impact  Status  Reports 

•  ESOH  Program  Progress  Reports 

Additionally,  the  database  has  the  capability  of 
generating  customized  reports  that  satisfy  the  needs 
of  both  the  system  safety  and  environmental 
communities. 
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Success  Recognized 


DoN  2006  Special  Recognition  for  Excellence  in  Safety  in  the  Field  of  Acquisition 


“The  program  emphasizes  the  integration  of  safety 
and  environmental  engineers  into  the  design/build 
teams  to  add  the  element  of  objectivity  into  hazard 
analyses.  This  team  exemplifies  the  benefits  of  the 
early  integration  of  safety  concerns  into  the 
acquisition  process.” 


tms  SECRETARY  OF  THE  NAVY 

¥¥A?N«w5r<?w  D  P  2PS&8  199? 

§r  2QC>$ 


MEMORANDUM  FOR  PROGRAM  MANAGER,  USS  OHIO  CLASS  88GN  PROGRAM 


SUBJECT:  Special  Recognition  for  Excellence  in  Safety  in  the  field  of  Acquisition 

The  Department  of  the  Navy  2(KKi  Special  Recognition  for  Excellence  in  Safety  in  the 
Field  of  Acquisition  is  presented  to  the  USS  OHIO  CLASS  SSGN  Program  (PMS.WX)  m 
recognition  of  rts  accomplishments  as  a  leader  m  promoting  acquisition  excellence  through 
effective-  risk  management  processes  and  hazard  recognition  and  cmrectknt. 

In  calendar  year  2005,  the  OHrO  Class  Environmental  Safety,  and  Occupational  Health 
(KSOH)  System  Safety  Integrated  Product  Team,  including  participants  from  Naval  Undersea 
Warfare  Center.  Newport  Division:  General  Dynamics.  Electric  Boat  Corporation;  and  Strategic 
Systems  Ihogroms  distinguished  itself  by  providing  early  identification,  elimination,  ami 
effective  control  of  system  safety  hazards  ami  environmental  impacts  associated  with  the  SSGN 
Conversion  Program. 

In  2005  the  team  closed  four  Weapon  Systems  Explosive  Safety  Review  Board 
(WSERB)  findings  and  completed  sixteen  analyses  and  reports  in  support  of  a  tiitte- compressed 
program  that  will  provide  exceptional  capability  to  the  Fleet,  In  recognition  of  their  exemplary 
performance  the  team  was  nominated  for  the  prestigious  David  Packard  Excellence  iu 
Acquisition  Award  in  the  category  of  supporting  Secretary  of  l>efeme  goals,  including  safety. 
Accomplishments  include: 

*  Making  Safety  the  responsibility  of  the  desigrVbuild  teams  to  ensure  application  of 
cognisant  engineering  expertise: 

*  Integrating  experienced  safety  and  environmental  engineer*  into  the  design-build 
teams  to  add  the  element  of  objectivity  into  hazard  analyses: 

*  Pefi ning  applicable  safety  hazards  and  environmental  impact?  and  defining  and  :he 
methods  of  elimination  or  mitigation  used  for  resolution;  and 

*  Incorporating  $$BN  experienced  engineers  into  the  design  buikl  teams  to  identify  and 
resolve  any  existing  safety  issues  and  to  implement  methods  of  mitigation  in  the- 
design  imd  manufacturing  of  the  SSGN. 

The  USS  OHIO  ("I -ASS  SSGN  Program  established  itself  as  a  leader  in  making  safety  an 
integral  part  of  its-  organisational  culture,  and  its  accomplishment*  are  in  keeping  with  the 
Department  of  the  Navy's  goals  and  objectives  for  professional  excellence. 

Coflgralufehpti?  on  a  job  well  done! 

Donald  C.  Winter 
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Program  Support  Review  Deep  Dive 


Pete  Nolte 

Systems  and  Software  Engineering 
Office  of  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology 

October  2007 
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What  Are  Program  Support  Reviews? 


USDfAT&Lf  Imperatives: 

•  “Provide  a  context  within  which  I  can  make 
decisions  about  individual  programs.” 

•  “Achieve  credibility  and  effectiveness  in  the 
acquisition  and  logistics  support  processes.” 

•  “Help  drive  good  systems  engineering 
practices  back  into  the  way  we  dc  ress. 


the  under  secretary  or  defense 
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d.  For  programs  where  I  am  the  MDA,  review  each  program’s  SEP  as  part 
or  the  preparation  for  Defense  Acquisition  Board  Milestone  Reviews  (DAB)  and 
other  acquisition  reviews,  provide  me  with  a  recommendation  on  the  program’s 
readiness  to  proceed  during  the  DAB.  Together  with  other  members  of  the  OSD 

staff,  lead  program  support  assessments  to  identify  and  help  resolve  issues  to 
ensure  program  success. 
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Systems  and  Software  Engineering 

Assessments  &  SuDoort 


Infrastructure  Support 


Ryan  Sinclair  Michelle  Grillo 
Beth  Bernat  (P/T)  Laura  Dwinnell  (P/T) 
Sarah  Rogers  (P/T) 


Land  Combat 


Pete  Nolte 

John  Mercer 
John  Quackenbush 
Jim  Waldeck  (P/T) 
Steve  Cox 


Fixed  Wing  Aircraft 


Jim  Thompson 

Bob  Darwin 
Scott  Menser 
Nicole  Bratt 
Bob  MacMullin 
Joe  D’Ambra 


Rotary  Wing  &  UAS 


Jim  Schultz 

Dick  Scott 
Kevin  Wilcutt 
James  Alexander 
Gregory  Carswell 


Communications 


Ken  Hong  Fong 

Regi  Chikar 
Jim  Wright 
Chuck  Johnson 
Steve  Raphael 


DEPUTY  DIRECTOR, 
ASSESSMENT  & 
SUPPORT 

Mr.  David  Castellano 


Glynn  James 

Suzette  Manduley 


Matrix 


Mike  Cribbs 
Lisa  Reuss 
Dave  Gallagher 
Spiros  Pallas 
Don  Gantzer 
Rich  Taylor 
Christine  Hines 


Peter  Tabbagh 
Jim  Bachana 
Mike  Zsak 
Donna  Carey 
Tom  Parry 
Chris  Powel  I 


Business 


Howard  Sterling 

Roger 

Kammerdeiner 
Steve  Hancock 


C2ISR 


Ray  Shanahan 
(Acting) 

Don  Maziarz 

Dick  Overmyer 


Subs  &  Ships 


Darren  Piccirillo 

Mike  Wagner 
John  Clifford 


Missiles 


Susan  van  der  Veer 

Doc  Holiday 
Gerry  Mello 


General  Review  Areas 


ASSESSMENT  METHODOLOGY  FOR  PRE-MILESTONE  C 


1.0 


2.0 


Mission  Capabilities/Requirements  Assessment  Area 
Sub-Area  1.1  -  Operational  Requirements 


4 
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ASSESSMENT  METHODOLOGY  FOR  PRE-MILESTONE  B 


1.0 


3.0 


2.0 


3.0 


4.0 


4.0 


5.0 


6.0 


5.0 


6.0 


Mission  Capabilities/Requirements  Assessment  Area  4 

Sub-Area  1.1  -  Operational  Requirements _ 4 


1.0 

ASSESSMENT  METHODOLOGY  FOR  PRE-MILESTONE  A 

Mission  Capabilities/Requirements  Assessment  Area 

4 

Sub-Area  1.1  -  Operational  Requirements 

4 

2.0 

Resources  Assessment  Area 

9 

Sub-Area  2.1  -  Program  Planning  and  Allocation 

9 

Sub-Area  2.2  -  Personnel 

10 

Sub-Area  2.3  -  Facilities 

12 

Sub-Area  2.4  -  Engineering  Tools 

13 

3.0 

Management  Assessment  Area 

16 

Sub-Area  3.1  -  Acquisition  Strategy/Process 

16 

Sub-Area  3.2  -  Project  Planning 

19 

Sub-Area  3.3  -  Program  and  Project  Management 

21 

Sub-Area  3.4  -  Contracting  and  Subcontracting 

26 

Sub-Area  3.5  -  Communication 

28 

4.0 

Technical  Process  Assessment  Area 

30 

Sub-Area  4.1  -  Technology  Assessment  and  Transition 

30 

Sub-Area  4.2  -  Requirements  Development 

31 

Sub-Area  4.3  -  Functional  Analysis  &  Allocation 

32 

Sub-Area  4.4  -  Design  Synthesis 

33 

Sub-Area  4.5  -  System  Integration,  Test  and  Verification 

35 

Sub-Area  4.6  -  Transition  to  Deployment 

37 

Sub-Area  4.7  -  Process  Improvement 

38 

5.0 

Technical  Product  Assessment  Area 

38 

Sub-Area  5.1  -  System  Description 

38 

Sub-Area  5.2  -  System  Performance 

42 

Sub-Area  5.3  -  System  Attributes 

43 

6.0 

Environment  Assessment  Area 

44 

Sub-Area  6.1  -  Statutory  and  Regulatory  Environment 

45 

h  ftp://www.acq.  osd.  mil/sse 
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Program  Support  Review  (PSR) 


•  DAPS;  a  repeatable,  tailorable,  exportable  process 

•  Trained  workforce  with  understanding  of  program  issues 


PSR  Reference  Matt’s 

•  Templates 

•  Sample  Questions 

•  Documented  Processes 

•  Training  Materials 

•  Execution  Guidance 


PSR  Evaluation  Areas 

1 .  Mission  Capabilities/ 
Requirements 

2.  Resources 

3.  Management 

4.  Technical  Process 

5.  Technical  Product 

6.  Environment 


‘...PSR  team  serves  as 
‘disinterested  3rd  party’  that 
allows  [the  PM]  to  approach 
leadership  armed  with 
powerful  program  truths, 
reinforce  issues.”  (PM) 


PMs  Report  Process  is  Insightful,  Valuable,  and  Results  Oriented; 
better  than  95%  acceptance  of  recommendations 
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Program  Support  Review  Activity 


(since  March  2004) 


PSRs/NARs  completed:  48 
AOTRs  completed:  1 1 
Nunn-McCurdy  Certification:  10 
Participation  on  Service-led  IRTs:  2 
Technical  Reviews:  10 
Reviews  planned  for  FY07: 

«  PSRs/NARs:  8 
«  AOTRs:  1 


Decision  Support  Reviews 


Pre-MS  C 
21% 


Nurm- 

McCurdy 

12% 


Pre-MSB 

31% 


DAE  Review 
7% 


Other 

19% 


Pre-MSA 

4% 


Service- Managed  Acquisitions 


Programs  by  Domain  Area 


Air  Force 
34% 


Marine 
Corps  9% 


Agencies 

9% 


Missiles  9% 


Navy 

20% 


Army 

28% 


Fixed  Wing 
19% 


Business  2% 
Space  5% 


Rotary  Wing 
17% 


Munitions  4% 


C2-ISR  9% 
Unmanned  5% 


Ships  7% 
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General  Approach:  Review  Products 


•  The  Team’s  top-level  products: 

-  Full  reviews  conducted  9-12  months  before  Milestone 

»  Detailed  findings,  risks  &  actionable  recommendations 
»  Conducted  in  “PM  support”  vice  “OSD  oversight”  mode 

-  “Quick-Look”  reviews  conducted  2-3  months  before  Milestone 

>>  Same  form  and  formats;  Conducted  “for  record”  review 

-  Quarterly  Defense  Acquisition  Executive  Summary  assessments 

-  Test  &  Evaluation  Master  Plan  (TEMP)  and  Systems  Engineering  Plan 
(SEP)  development  and  approval 
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Findings 


Program  Support  Review 
Taxonomy  of  Classifications 


+  Positive 
O  Neutral 
-  Negative 
»"  Issue 
-  Risk 


+  Positive 

w 

O  Neutral 

— ► 

May  be  a  candidate  for  Best  Practice 


Current  focus 
of  Systemic 
Analysis 


May  be  a  candidate  for  Process  Improvement  Recommendation 


-  Negative 


-  Risk 


Issue 


Potential 


Root  Cause(s) 

Impact(s) 

Recommendation(s) 

Root  Cause(s) 

Impact(s) 

Recommendation(s) 

i  -  Risk 

i 

- ► 

Root  Cause(s) 

Impact(s) 

Recommendation(s) 
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Program  Support  Review  Findings 

(March  2004  Through  September  2007) 


Level  Two  DAPS  Methodology  Areas 


Does  not  include  findings  associated 
with  the  new  DAPS  methodology 


□  First  31  Programs  ■  Next  20  Programs 


Level  Two  Methodology  Codes  ■  48  PSRs,  TRs,  &  NM  Reviews 
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Representative  PSR  Findings  (1  of  3) 


1.1  Mission  Capabilities/Requirements 

-  Lack  of  reasonable,  measurable,  and  testable  requirements 

-  Requirements  refer  to  “predecessor”  systems 

-  Requirements  changes  contribute  to  SE  churn 

-  Lingering  requirements  issues  have  increased  program  costs  and  risks 

-  Failure  to  establish  a  process  for  flowing  down  requirements 

-  Requirements  are  not  fully  understood  after  contract  award 

-  Lack  of  growth  margins/trade-space 

3.1  Acquisition  Strategy/Process 

-  Resistance  to  demonstrate  key  functionality  by  MS  C 

-  Balance  between  requirements,  schedule  and  resources 

-  Acquisition  strategy  doesn’t  address  key  issues 


Representative  PSR  Findings  (2  of  3) 


3.2  Project  Planning 

-  Schedule  vs.  event  driven  programs 

-  No  “time”  to  conduct  the  full  suite  of  SE  technical  reviews 

-  Lack  of  Integrated  Master  Plan/Integrated  Master  Schedule 

-  Underestimation  of  integration  efforts  and  COTS  modifications 

-  Lack  of  meaningful  acquisition  phase  exit  criteria 

3.3  Program  &  Project  Management 

-  Marginal  Program  Office  staffing;  Difficult  to  retain  high  quality  personnel 

-  Roles,  responsibilities,  and  lines  of  authority  are  not  clear 

-  Poor  communication  across  IPTs  and  program  lines 

-  Lack  of  management  metrics  to  monitor  program  health 

-  EVMS  does  not  provide  insight  and  does  not  reflect  work  being  done 

-  Lack  of  properly  documented  risks  and  mitigation  plans 
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Representative  PSR  Findings  (3  of  3) 


4.5  System  Integration,  Test,  &  Verification 


-  Highly  concurrent  test  schedules;  Success-oriented 

-  Aggressive  schedule  lacks  adequate  time  for  corrective  actions 

-  Optimistic  plans  to  leverage  M&S;  Lack  of  VV& A  planning 

-  Shortage  of  military  operators  for  operational  tests 

-  Testing  and  verification  approach  are  inadequate 

-  Developmental  testing  not  complete  prior  to  IOT&E 


5.3  System  Attributes 

-  Insufficient  efforts  to  design-in  reliability  and  maintainability,  including  diagnostics 

-  Weak  emphasis  on  suitability  contributes  to  IOT&E  issues 

-  Late  production  planning;  Insufficient  Production  Readiness  Reviews 

-  Challenging  production  ramp  rates  for  contractors/suppliers 

-  Optimistic  software  productivity,  reuse  and  growth  estimates 
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Thoughts  That  Need  Reinforcement  (1  of  3) 


•  Mission  Capabilities/Requirements 

-  Ensure  CDD/CPD  requirements  are  reasonable,  measurable  and  testable 

-  Ensure  approved  CONOPS  informs  requirements  generation  process 

-  Maintain  stable  requirements 

-  Conduct  cost/performance  trades  with  PM,  user  and  contractors 

-  Push  high  risk  requirements  to  the  next  increment 

-  Conduct  SRR  in  TD  phase  with  contractors 

-  Understand  COTS/GOTS  capabilities  and  limitations  (when  operated  in  a  military 
environment) 

-  Be  aware  of  critical  dependence  on  external  programs  with  developmental  issues 

-  Establish  space/weight/power/cooling  margins 

•  Management 

-  Balance  requirements,  resources  and  acquisition  strategy 

-  Plan  to  demonstrate  key  functionality  in  SDD  phase 

-  Maintain  event  driven  schedules;  establish  entry/exit  criteria 

-  Use  earned  value  management  as  a  vehicle  for  planning,  executing,  and 
controlling  the  program 

-  Employ  a  robust  risk  management  process  and  resource  mitigation  activities 
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Thoughts  That  Need  Reinforcement  (2  of  3) 


•  Management  (cont.) 

-  Ensure  communication  between  IPTs;  and  with  Contractor 

-  Define  IPT  roles,  responsibilities,  authority  and  conflict  resolution  process 

-  Manage  external  interfaces;  establish  issue  resolution  process 

-  Avoid  urgency  of  need  outweighing  good  engineering  and  program  management 

•  Resources 

-  Ensure  funding  is  properly  phased  and  adequate  to  support  planned  SE  activities 

-  Adequately  staff  the  program  with  qualified  personnel 

-  Ensure  early  selection  of  M&S  and  plan  to  VV&A  planning 

-  Ensure  adequate  management  reserve 

•  Technical  Product 

-  Use  mature  technologies  and  modular  open  architecture 

-  Assess  COTS/GOTS  form  factor  changes  and  integration  challenges 

-  Plan  to  design-in  reliability  and  maintainability 

-  Assess  supportability  in  the  SDD  phase 

-  Provide  early  focus  on  production  planning 

-  Use  realistic  software  size,  productivity,  and  reuse  estimates 

-  Ensure  test  schedule  reflects  adequate  time  for  corrective  actions  and  reporting 
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Thoughts  That  Need  Reinforcement  (3  of  3) 


•  Technical  Process 


-  Use  established  SE  processes 

»  Full  suite  of  SE  technical  reviews 
»  Independent  chairman  and  SMEs 
»  Adequate  time  between  technical  reviews/SDD  events 
»  Maintain  technical  baselines 
»  Process  compliance 

-  Ensure  translation  of  operational  requirements  into  contractual  language 

-  Comprehensive  contractual  verification  (section  4  of  spec)  of  meeting 
requirements  (section  3  of  spec) 

-  Ensure  adequate  requirements  flow-down/  traceability/  decomposition 

-  Put  emphasis  on  test  and  verification  approach 


•  Environment 

-  Ensure  consistency  in  program  documentation 

-  Be  aware  of  new  policies,  Congressional  language,  and  certifications 
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Questions... perhaps  Answers 
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Back-up  Slides 
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Samples  of  Program  Support  Review 
Positive  Observations 


•  Experienced  and  dedicated  program  office  teams 

•  Strong  teaming  between  PM  offices  and  contractors 

•  Use  of  well  defined  and  disciplined  SE  processes 

•  Proactive  use  of  independent  review  teams 

•  Successful  management  of  external  interfaces 

•  Corporate  commitment  to  process  improvement 

•  Notable  manufacturing  processes 

•  Appropriate  focus  on  performance-based  logistics 

•  Focus  on  DoD  initiatives 

•  Excellent  risk  management  practices 


But  not  on  all  Programs. . . 
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